Control device, control method, control system, and non-transitory recording medium

ABSTRACT

A control device acquires a plurality of images including an ultrasound image and a photoacoustic image taken in an examination, selects a first image and a second image to be displayed superimposed on the first image, based on information relating to the examination, and generates an object for outputting at least information for displaying the second image superimposed on the first image to an external device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of International Patent ApplicationNo. PCT/JP2017/034989, filed Sep. 27, 2017, which claims the benefit ofJapanese Patent Application No. 2016-198893, filed Oct. 7, 2016, both ofwhich are hereby incorporated by reference herein in their entirety.

TECHNICAL FIELD

The present invention relates to a control device, a control method, acontrol system, and a program.

BACKGROUND ART

Ultrasound imaging devices and photoacoustic imaging devices are used asimaging devices for minimally-invasive imaging of states within the bodyof a subject of examination. For example, PTL 1 discloses generating asuperimposed image where a photoacoustic image is superimposed on anultrasound image in diagnosis using medical images.

CITATION LIST Patent Literature

PTL 1 Japanese Patent Laid-Open No. 2005-21380

In a case where a superimposed image is generated at a certain device,and multiple medical images included in the superimposed image aretransmitted to a viewer provided outside of this device, the externalviewer has no information on how to superimpose the multiple medicalimages, and accordingly cannot generate a superimposed image. That is tosay, there has been a problem in that some viewers cannot generatedsuperimposed images.

SUMMARY OF INVENTION

A control device according to an embodiment of the present inventionincludes acquiring means configured to acquire a plurality of imagesincluding an ultrasound image and a photoacoustic image taken in anexamination, selecting means configured to select, from the acquiredplurality of images, a first image, and a second image to be displayedsuperimposed on the first image, based on information relating to theexamination, and generating means configured to generate an object foroutputting, to an external device, at least information for displayingthe second image superimposed on the first image.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of the functionalconfiguration of a control device according to an embodiment of thepresent invention.

FIG. 2 is a diagram illustrating an example of the hardwareconfiguration of the control device according to the embodiment of thepresent invention.

FIG. 3 is a diagram illustrating an example of information generated bythe control device according to the embodiment of the present invention.

FIG. 4 is a flowchart illustrating an example of processing performed bythe control device according to the embodiment of the present invention.

FIG. 5 is a flowchart illustrating an example of processing performed bythe control device according to the embodiment of the present invention.

FIGS. 6A and 6B are a flowchart illustrating an example of processingperformed by the control device according to the embodiment of thepresent invention.

FIG. 7 is a diagram illustrating an example of a screen displayed on adisplay unit by the control device according to the embodiment of thepresent invention.

FIG. 8A is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 8B is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 8C is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 8D is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 8E is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 8F is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 8G is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 8H is a diagram illustrating an example of a display form of amedical image by the control device according to the embodiment of thepresent invention.

FIG. 9 is a diagram illustrating an example of a screen displayed on adisplay unit by the control device according to the embodiment of thepresent invention.

FIG. 10 is a diagram illustrating an example of a screen displayed on adisplay unit by the control device according to the embodiment of thepresent invention.

FIG. 11 is a diagram illustrating an example of information generated bythe control device according to the embodiment of the present invention.

FIG. 12A is a diagram illustrating an example of information generatedby the control device according to the embodiment of the presentinvention.

FIG. 12B is a diagram illustrating an example of information generatedby the control device according to the embodiment of the presentinvention.

FIG. 13 is a diagram illustrating an example of a screen displayed on adisplay unit by the control device according to the embodiment of thepresent invention.

FIG. 14A is a diagram illustrating an example of a screen displayed on adisplay unit by the control device according to the embodiment of thepresent invention.

FIG. 14B is a diagram illustrating an example of a screen displayed on adisplay unit by the control device according to the embodiment of thepresent invention.

FIG. 15 is a diagram illustrating an example of the configuration of asystem including the control device according to the embodiment of thepresent invention.

DESCRIPTION OF EMBODIMENTS

An embodiment of the present invention will be described below withreference to the drawings. In the present specifications, acoustic wavesgenerated by expansion occurring in the body of a subject of examinationdue to the subject body being irradiated by light are referred to asphotoacoustic waves. Also, acoustic waves transmitted from a transducer,and reflected waves (echo) where the transmitted acoustic waves arereflected in the body of the subject of examination, are referred to asultrasound waves.

Imaging methods using ultrasound and techniques of imaging usingphotoacoustic waves are being used as methods for minimally-invasiveimaging of states inside the body of a subject of examination. Themethod of imaging using ultrasound is a method where ultrasound wavesemitted from a transducer, for example, are reflected at tissues insidethe body of the subject due to difference in acoustic impedance, and animage is generated based on the time for reflected waves to arrive atthe transducer, and the intensity of the reflective waves. An image madeby imaging using ultrasound waves will be referred to hereinafter as anultrasound image. The user operates a probe while changing the angle andso forth, and can observe ultrasound images of various types ofcross-sections in real-time. Ultrasound images visualize shapes oforgans and tissue, and is being used to discover tumors and so forth.The method of imaging using photoacoustic waves is a method where animage is generated based on photoacoustic waves generated by thermalexpansion of tissue within the body of the subject that has beenirradiated by light, for example. An image where imaging has beenperformed using photoacoustic waves will be referred to hereafter as aphotoacoustic image.

Photoacoustic images visualize information relating to opticalproperties, such as the degree of absorption of light by the tissues. Itis known that blood vessels can be visualized by photoacoustic imagesdue to optical properties of hemoglobin, for example, and studies arebeing made for use in evaluation of malignancy of tumors or the like.

There are cases where various types of information are collected byimaging the same part of the body of a subject based on differentprinciples, in order to improve the accuracy of diagnosis. For example,there are cases where diagnosis regarding cancer is performed bycombining form information obtained by computed tomography (CT) imagesand functional information related to metabolism obtained frompositron-emission tomography (PET) images. It is thought that performingdiagnosis using information obtained by imaging different phenomenabased on different principles is effective in improving accuracy ofdiagnosis.

Imaging devices for obtaining images that combine the properties of theabove-described ultrasound images and photoacoustic images are beingstudied. Particularly, both ultrasound images and photoacoustic imagesare imaged using ultrasound waves from the body of the subject, soimaging of ultrasound images and photoacoustic images can be performedusing the same imaging device. More specifically, reflected waves thatthe subject body has been irradiated by and photoacoustic waves can bereceived at the same transducer. Accordingly, ultrasound wave signalsand photoacoustic signals can be acquired at a single probe, so animaging device that performs imaging of ultrasound images and imaging ofphotoacoustic images can be realized with the same device, without thehardware configuration becoming complicated.

For example, in a case of performing examination using an imaging devicethat performs imaging of ultrasound images and imaging of photoacousticimages as described above, it is envisioned that various images will begenerated during a single examination. An example of a case will bedescribed where imaging of ultrasound images is being performed, and inconjunction with this, imaging of photoacoustic images is performed bythe user performing input of instructions instructing irradiation bylight as appropriate. A moving image made up of a group of a series ofultrasound images, or ultrasound images at timings instructed by theuser, are generated in a single examination. Each time the body of thesubject is irradiated by light in a single examination, photoacousticwaves are generated in the body, and various types of photoacousticimages are generated which will be described later. In a case whereirradiation by light is performed multiple times in a singleexamination, multiple photoacoustic images can be generated at eachtiming of irradiation by light.

In a case where images are transmitted to an external device asindividual objects, there is concern that superimposed display cannot bemade depending on the specifications of the viewer, and further, theuser has to set the form of the superimposed display manually.Alternatively, in a case where a composited image in which multipleimages have been composited is transmitted to the external device as asingle object, the user can observe in a state where the multiple imageshave been composited, but cannot display each of the multiple imagesindividually on the viewer.

In light of these problems, it is preferable, from the perspective ofperforming observation while comparing ultrasound images andphotoacoustic images, to store ultrasound images and photoacousticimages imaged at generally the same time in a correlated manner.Further, at the time of outputting a correlated image group to anexternal device, it is preferable that an automatically displayableobject be generated and output in a form suitable for the user.

A control device according to an embodiment of the present invention isprovided to improve the workflow efficiency of interpreting anddiagnosing, for example, by generating an object at an external devicewhere multiple images can be displayed to the user in a suitable form.

FIG. 15 is a diagram illustrating an example of the configuration of asystem including a control device 101 according to the embodiment of thepresent invention. The imaging system 100 that is capable of generatingultrasound images and photoacoustic images is connected to various typesof external devices via a network 110. The configurations included inthe imaging system 100 and the various types of external devices do nothave to be installed within the same facilities, and it is sufficientfor these to be communicably connected.

The imaging system 100 includes a control device 101, a probe 102, adisplay unit 104, and an operating unit 105. The imaging control device101 is a device that acquires ultrasound wave signals and photoacousticsignals from the probe 102, control acquisition of the photoacousticsignals based on the ultrasound signals for example, and generatesphotoacoustic images based on this control. The control device 101 alsoacquires information relating to examinations including ultrasoundimages and photoacoustic images from an ordering system 112, andcontrols the probe 102 and display unit 104 when the examination isperformed. The control device 101 outputs generated ultrasound images,photoacoustic images, and superimposed images where photoacoustic imageshave been superimposed on ultrasound images, to a picture archiving andcommunication system (PACS) 113. The control device 101 exchangesinformation with external devices such as the ordering system 112 andPACS 113, in accordance with standards such as Health Level 7 (HL7) andDigital Imaging and Communications in Medicine (DICOM). Details ofprocessing performed by the control device 101 will be described later.

Examples of regions in the body of the subject of examination regardingwhich ultrasound images are to be taken by the imaging system 100include the cardiovascular region, breasts, liver, pancreas, abdomen,and so forth. The imaging system 100 may also take ultrasound images ofthe subject body where a microbubble-based ultrasound contrast agent hasbeen administered, for example.

Examples of regions in the body of the body of the subject ofexamination regarding which photoacoustic images are to be taken by theimaging system 100 are regions such as the cardiovascular region,breasts, cervical region, abdomen, the extremities including fingers andtoes, and so forth. Particularly, new blood vessels and blood vesselregions including plaque on vascular walls may be the object of takingphotoacoustic images, in accordance with characteristics regarding lightabsorption within the subject body. Although an example will bedescribed below regarding a case of taking photoacoustic images whiletaking ultrasound image, the region on the body of the subject wherephotoacoustic images are being taken by the imaging system 100 does notnecessarily have to agree with the region where ultrasound images arebeing taken. Dyes such as methylene blue, indocyanine green, fine goldparticles, and substances where these have been aggregated or chemicallymodified may be used as a contrast agent in the imaging system 100, andadministered to the subject for imaging of photoacoustic images.

The probe 102 is operated by the user and transmits ultrasound wavesignals and photoacoustic signals to the control device 101. The probe102 includes a transmission/reception unit 106 and an irradiation unit107. The probe 102 transmits ultrasound waves from thetransmission/reception unit 106, and receives the reflected waves at thetransmission/reception unit 106. The probe 102 irradiates the subject bylight from the irradiation unit 107, and receives the photoacousticwaves at the transmission/reception unit 106. The probe 102 converts thereceived reflected waves and photoacoustic waves into electricalsignals, and transmits to the control device 101 as ultrasound wavesignals and photoacoustic signals. The probe 102 is preferablycontrolled so as to execute transmission of ultrasound waves foracquiring ultrasound wave signals and irradiation by light to acquirephotoacoustic signals when information is received that contact has beenmade with the subject body.

The transmission/reception unit 106 includes at least one transducer(omitted from illustration), a matching layer (omitted fromillustration), a damper (omitted from illustration), and an acousticlens (omitted from illustration). The transducer (omitted fromillustration) is formed of a material that exhibits the piezoelectriceffect, such as lead zirconate titanate (PZT) or polyvinylidenedifluoride (PVDF). The transducer (omitted from illustration) may beother than a piezoelectric element, and may be a capacitive transducer(capacitive micro-machined ultrasonic transducer (CMUT)) or a transducerusing a Fabry-Perot interferometer, for example. Typically, ultrasoundwave signals are 2 to 20 MHz and photoacoustic signals are 0.1 to 100MHz in frequency component, so an arrangement that can detect thesefrequencies is used for the transducer (omitted from illustration). Thesignals obtained by the transducer (omitted from illustration) aretime-division signals. The amplitude of the received signals representsvalues based on acoustic pressure received at the transducer at eachtime. The transmission/reception unit 106 includes a circuit (omittedfrom illustration) for electronic focus or a control unit. The layout ofthe transducer (omitted from illustration) is a sector array, lineararray, a convex array, an annular array, or a matrix array, for example.The probe 102 acquires ultrasound wave signals and photoacousticsignals, which may be acquired alternately, or acquired simultaneouslyor may be acquired according to a preset form.

The transmission/reception unit 106 may have an amplifier (omitted fromillustration) that amplifies time-sequence analog signals that thetransducer (omitted from illustration) has received. Thetransmission/reception unit 106 may also have an A/D converter thatconvers time-sequence analog signals that the transducer (omitted fromillustration) has received into time-sequence digital signals. Thetransducer (omitted from illustration) may be divided into atransmitting portion and a receiving portion in accordance with thepurpose of imaging the ultrasound image. The transducer (omitted fromillustration) may also be divided into a portion for imaging ultrasoundimages and a portion for imaging photoacoustic images.

The irradiation unit 107 includes a light source (omitted fromillustration) for acquiring photoacoustic signals and an optical system(omitted from illustration) that guides pulse light emitted from thelight source (omitted from illustration) to the subject. The pulse widthof light emitted from the light source (omitted from illustration) is apulse width of 1 ns or longer to 100 ns or shorter, for example. Thewavelength of the light that the light source (omitted fromillustration) emits is a wavelength of 400 nm or longer to 1600 nm orshorter, for example. In a case of imaging blood vessels near thesurface of the subject with high resolution, a wavelength of 400 nm orlonger to 700 nm or shorter, where the absorption at blood vessels isgreat, is preferable. In a case of imaging deep portions of the body ofthe subject, a wavelength of 700 nm or longer to 1100 nm or shorter,that is not readily absorbed by water or tissue such as fat, ispreferable.

The light source (omitted from illustration) is a laser orlight-emitting diode, for example. A light source where wavelength canbe changed may be used as the irradiation unit 107, in order to acquirephotoacoustic signals using light of multiple wavelengths.Alternatively, the irradiation unit 107 may be of a configuration havingmultiple light sources that emit light of different wavelengths fromeach other, where irradiation can be alternately performed by light ofdifferent wavelengths from the different light sources. Examples oflaser include solid-state laser, gas laser, dye laser, and semiconductorlaser. A pulsed laser such as an Nd:YAG laser or an alexandrite lasermay be used for the light source (omitted from illustration). Also, aTi-sapphire laser or optical parametric oscillator (OPO) laser that usesNd:YAG laser light as excitation light may be used as the light source(omitted from illustration). Further, a microwave source may be used asthe light source (omitted from illustration).

Optical elements such as lenses, mirrors, optical fibers, and so forth,are used as the optical system (omitted from illustration). Irradiationis preferably performed with the beam diameter of the pulsed lightexpanded in a case where the subject is a breast, so a diffraction platefor diffracting the emitted light may be provided to the optical system(omitted from illustration). Alternatively, a configuration may be madewhere the optical system (omitted from illustration) has lenses or thelike and can carry out beam focusing, in order to raise resolution.

The display unit 104 displays images imaged by the imaging system 100and information relating to the examination, based on control from thecontrol device 101. The display unit 104 provides an interface to acceptuser instructions, based on control from the control device 101. Anexample of the display unit 104 is a liquid crystal display.

The operating unit 105 transmits information relating to input of useroperations to the control device 101. The operating unit 105 is, forexample, a keyboard or trackball, or various types of buttons forperforming input of operations relating to examination.

The display unit 104 and operating unit 105 may be integrated as a touchpanel display. There is no need for the control device 101, display unit104, and operating unit 105 to be separate devices, and theseconfigurations may be realized as an integrated console. The controldevice 101 may have multiple probes.

A hospital information system (HIS) 111 is a system that supportshospital operations. The HIS 111 includes an electronic health recordsystem, an ordering system, and an medical accounting system. The HIS111 enables comprehensive management from ordering examinations up toaccounting. The ordering system of the HIS 111 transmits orderinformation to ordering systems 112 of respective departments.Implementation of the order is managed at the ordering system 112described later.

The ordering system 112 is a system that manages examinationinformation, and manages progress of each examination at the imagingdevice. An ordering system 112 may be configured for each departmentthat performs examination. An example of the ordering system 112 is, inthe case of a radiology department, a radiology information system(RIS). The ordering system 112 transmits information for examinations tobe performed at the imaging system 100 to the control device 101, inresponse to a query from the control device 101. The ordering system 112receives information relating to progress of examinations from thecontrol device 101. Upon having received information from the controldevice 101 to the effect that an examination has been completed, theordering system 112 transmits information indicating that thisexamination has been completed to the HIS 111. The ordering system 112may be integrated with the HIS 111.

A picture archiving and communication system (PACS) 113 is a databasesystem that stores images obtained at various types of imaging devicesinside of and outside of the facilities. The PACS 113 has a storage unit(omitted from illustration) that stores medical images and imagingconditions of the medical images, supplementary information such asparameters of image processing including reconfiguration, patientinformation, and so forth, and a controller (omitted from illustration)that manages information stored in this storage unit. The PACS 113stores ultrasound images, photoacoustic images, and superimposed images,which are objects output form the control device 101. Communicationbetween the PACS 113 and the control device 101, and various types ofimages stored in the PACS 113, preferably conform to standards such asHL7 and DICOM. Various types of images output from the control device101 are correlated with supplementary information in various types oftags in accordance with the DICOM standard, and stored.

A viewer 114 is an image diagnosis terminal, that reads out imagesstored in the PACS 113 or the like, and displays for diagnosis. Aphysician displays images on the viewer 114 and observes the images, andrecords information obtained as a result of this observation as an imagediagnosis report. Image diagnosis reports created using the viewer 114may be stored in the viewer 114, or may be output to the PACS 113 or areport server (omitted from illustration) and stored.

A printer 115 prints images stored in the PACS 113 or the like. Anexample of the printer 115 is a film printer, that outputs images storedin the PACS 113 or the like by printing on film.

FIG. 2 is a diagram illustrating an example of the configuration of thecontrol device 101. The control device 101 includes a central processingunit (CPU) 201, read-only memory (ROM) 202, random access memory (RAM)203, a hard disk drive (HDD) 204, a Universal Serial Bus (USB) port 205,a communication circuit 206, a graphics processing unit (GPU) 207, aHigh-Definition Multimedia Interface (HDMI, a registered trademark) port208, and a probe connector port 210. These components are communicablyconnected by a bus. The bus is a database that is used for exchangingdata among the hardware connected thereto, and for transmitting commandsfrom the CPU 201 to other hardware.

The CPU 201 is a control circuit that centrally controls the controldevice 101 and components connected thereto. The CPU 201 carries outcontrol by executing programs stored in the ROM 202. The CPU 201 alsoruns a display driver that is software for controlling the display unit104, and performs display control of the display unit 104. The CPU 201further performs input/output control of the operating unit 105.

The ROM 202 stores programs and data recording procedures for control bythe CPU 201. The ROM 202 stores a boot program for the control device101 and various types of initialization data. The ROM 202 also storesvarious types of programs for realizing the processing by the controldevice 101.

The RAM 203 provides a workspace storage region for the CPU 201 toperform control by control programs. The RAM 203 has stack and workregions. The RAM 203 stores programs that the control device 101 andcomponents connected thereto executing the processing of, and varioustypes of parameters used in image processing. The RAM 203 stores controlprograms executed by the CPU 201, and temporarily stores various typesof data for the CPU 201 to execute various types of control.

The HDD 204 is an auxiliary storage device for saving various types ofdata, such as ultrasound images, photoacoustic images, and so forth.

The USB port 205 is a connector to connect to the operating unit 105.

The communication circuit 206 is a circuit for performing communicationwith components making up the imaging system 100, and various types ofexternal devices connected to the network 110. The communication circuit206 stores information to be output in the form of transmission packets,and outputs to external devices via the network 110 by communicationtechnology such as TCP/IP, for example. The control device 101 may havemultiple communication circuits in accordance with desired communicationforms.

The GPU 207 is included in a general-purpose graphics board having videomemory. The GPU 207 performs reconstruction processing of photoacousticimages, for example. Using this sort of processing unit enablesprocessing such as reconstruction to be performed at high speeds withoutrequiring dedicated hardware.

The HDMI (registered trademark) port 208 is a connector for connectingto the display unit 104.

The probe connector port 210 is a port for connecting the probe 102 tothe control device 101. The ultrasound wave signals and photoacousticsignals output from the probe 102 are acquired by the control device 101via the probe connector port 210.

The CPU 201 and GPU 207 are examples of a processor. The ROM 202, RAM203, and HDD 204 are examples of memory. The control device 101 may havemultiple processors. In a first embodiment, the functions of componentsof the control device 101 are realized by the processor of the controldevice 101 executing programs stored in memory.

Note that the control device 101 may have a CPU or GPU that performsparticular processing in a dedicated manner. The control device 101 mayalso have a field-programmable gate array (FPGA) where particularprocessing or all processing has been programmed. The control device 101may have a solid state drive (SSD) as an auxiliary storage device.

FIG. 1 is a diagram illustrating an example of the functionalconfiguration of the control device 101. The control device 101 includesan examination implementation information saving unit 120, an imageprocessing unit 121, a imaging control unit 122, a signal acquisitionunit 123, a settings saving unit 124, an examination control unit 125,an input/output control unit 126, an external transmission informationgenerating unit 127, and a transmission/reception control unit 128.

The examination implementation information saving unit 120 storesinformation of examinations carried out in the past. Examinationinformation stored in the examination implementation information savingunit 120 is registered, updated, deleted, and searched, based on inputof user operations and control from configurations related to thecontrol device 101. The examination implementation information savingunit 120 is made up of a database.

The image processing unit 121 generates ultrasound images andphotoacoustic images, and superimposed images where photoacoustic imageshave been superimposed on ultrasound images. The image processing unit121 generates ultrasound images to be displayed on the display unit 104from ultrasound wave signals acquired from the signal acquisition unit123. The image processing unit 121 generates ultrasound imagesappropriate for the mode that has been set, based on information ofimaging procedures acquired from the examination control unit 125. Forexample, in a case where the Doppler mode is set as the imagingprocedure, the image processing unit 121 generates an image indicatingflow rate within the subject body, based on the difference between thefrequency of ultrasound wave signals acquired by the signal acquisitionunit 123 and the transmission frequency.

The image processing unit 121 also generates photoacoustic images basedon photoacoustic signals acquired by the signal acquisition unit 123.The image processing unit 121 reconstructs distribution of photoacousticwaves when irradiated by light (hereinafter referred to as initialacoustic pressure distribution), based on photoacoustic signals. Theimage processing unit 121 acquires a distribution of coefficients oflight absorption in the subject body, by dividing the initial acousticpressure that has been reconstructed by the light fluence distributionin the subject body of the light that the subject body has beenirradiated by. A concentration distribution of matter in the subjectbody is also acquired from distribution of coefficients of lightabsorption as to multiple wavelengths, using the fact that the degree ofabsorption of light within the body differs in accordance with thewavelength of light that the body is irradiated by. For example, theimage processing unit 121 acquires the distribution of matter in thesubject body regarding oxyhemoglobin and deoxyhemoglobin. The imageprocessing unit 121 further acquires oxygen saturation distribution as aratio of oxyhemoglobin concentration as to deoxyhemoglobinconcentration. Photoacoustic images generated by the image processingunit 121 are images indicating information such as the above-describedinitial acoustic pressure distribution, light fluence distribution,absorption coefficient distribution concentration distribution ofmatter, and oxygen saturation distribution.

The image processing unit 121 further composites data of multipleimages, and generates a single composite image. For example, an imagewhere a photoacoustic image has been superimposed on an ultrasound imageis generated as a composite image. Hereinafter, an image that serves asthe background of a superimposed display will be referred to as a baseimage, and an image superimposed on the base image as a layer image. Theimage that is being displayed superimposed will be referred to as asuperimposed image. The image processing unit 121 performs imageprocessing for display and diagnosis assistance on the image data, suchas gradient processing. The image processing unit 121 is an example ofacquisition means for acquiring ultrasound images and photoacousticimages.

The imaging control unit 122 controls the probe 102 based on informationof imaging procedures received from the examination control unit 125.The imaging control unit 122 transmits information relating to theexamination order including information of the imaging procedure to thesignal acquisition unit 123. The imaging control unit 122 controls theflow relating to acquisition of ultrasound wave signals and acquisitionof photoacoustic signals in the examination.

The signal acquisition unit 123 obtains ultrasound wave signals andphotoacoustic signals from the probe 102. Specifically, the signalacquisition unit 123 acquires ultrasound wave signals and photoacousticsignals from information acquired from the probe 102, whiledistinguishing between the two, based on information from theexamination control unit 125 and the imaging control unit 122. Forexample, there are cases where the timing of acquisition of ultrasoundwave signals and photoacoustic signals is stipulated in the imagingprocedure being used for imaging. In this case, the signal acquisitionunit 123 acquires ultrasound wave signals and photoacoustic signals frominformation acquired from the probe 102, based on the acquisition timinginformation acquired from the examination control unit 125, whiledistinguishing between the two.

The settings saving unit 124 stores information of setting relating toactions of the control device 101, such as imaging procedure information301, failed shot information, and so forth. For example, the settingssaving unit 124 accepts and stores setting relating to generating of anobject for superimposed display of image data acquired by the imagingsystem 100 at an external device. The information of the settings storedin the settings saving unit 124 is registered updated, deleted, andsearched, based on input of user operations and control fromconfigurations related to the control device 101. Details of informationof settings stored in the settings saving unit 124 will be describedlater. The settings saving unit 124 is made up of a database. Thesettings saving unit 124 is an example of accepting means.

The examination control unit 125 controls examinations performed by theimaging system 100. The examination control unit 125 acquiresinformation of examination orders from the ordering system 112.Examination orders include information of the patient to be examined,and information relating to the imaging procedure. The examinationcontrol unit 125 transmits information relating to the examination orderto the imaging control unit 122. The examination control unit 125 alsodisplays information of the examination on the display unit 104 via adisplay control unit (omitted from illustration), to present informationrelating to the examination to the user. Information of the examinationthat is displayed on the display unit 104 includes information of thepatient to be examined, information of imaging procedures involved inthe examination, and images that have already been imaged and generated.The examination control unit 125 further transmits information relatingto the progress of the examination to the ordering system 112. Forexample, when the examination is started by the user, the examinationcontrol unit 125 notifies the ordering system 112 of the start, and whenimaging by all imaging procedures included in the examination iscompleted, notifies the ordering system 112 of the completion.

The input/output control unit 126 controls the display unit 104 todisplay information on the display unit 104. The input/output controlunit 126 displays information on the display unit 104 in accordance withinput from the examination control unit 125 and image processing unit121, and input of user operations via the operating unit 105. Theinput/output control unit 126 also acquires information from theoperating unit 105.

The external transmission information generating unit 127 generatesinformation for transmitting various types of information to externaldevices such as the PACS 113 and viewer 114, in accordance with controlfrom the examination control unit 125 and input of user operations. Forexample, the external transmission information generating unit 127generates information to output ultrasound images and photoacousticimages generated at the image processing unit 121, and superimposedimages thereof, to the PACS 113. In formation output to external devicesinclude supplemental information supplemented as various types of tagsfollowing the DICOM standard. The supplemental information includespatient information, information indicating the imaging device that hasimaged this image, image ID for uniquely identifying the image, andexamination ID for uniquely identifying the examination where the imagewas imaged. The supplemental information also includes informationcorrelating ultrasound images and photoacoustic images imaged during theexamination. Information correlating ultrasound images and photoacousticimages is information indicating, out of multiple frames making upultrasound images for example, a frame that is the closest to the timingof having acquired a photoacoustic image. That is to say, the externaltransmission information generating unit 127 generates objects fortransmission to external devices based on information stored in theexamination implementation information saving unit 120 and settingssaving unit 124, and the ultrasound images and photoacoustic imagesacquired at the image processing unit 121. Objects are information thatare the object of transmission from the control device 101 to externaldevices such as the PACS 113 and viewer 114. Objects generated by theexternal transmission information generating unit 127 includeinformation for displaying a certain image superimposed on anotherimage, at least. There are cases where the object is configured of imagedata and supplemental information regarding that image data. Theexternal transmission information generating unit 127 generates, forexample, external transmission information 1101. Details of externaltransmission information 1101 will be described later. The externaltransmission information generating unit 127 is an example of generatingmeans. The external transmission information generating unit 127 is alsoan example of selecting means.

The transmission/reception control unit 128 controls transmission andreception of information among external device such as the orderingsystem 112, PACS 113, and viewer 114, and the control device 101, viathe network 110. The transmission/reception control unit 128 receivesinformation of examination orders from the ordering system 112. Thetransmission/reception control unit 128 transmits objects generated atthe external transmission information generating unit 127 to the PACS113 and viewer 114.

Note that FIGS. 1, 2, and 15 illustrate an example where the controldevice 101 is connected to the probe 102 and controls imaging ofultrasound images and photoacoustic images, but the control deviceaccording to an embodiment of the present invention is not necessarilyrestricted to this form. The control device according to the embodimentof the present invention may have a configuration where ultrasoundimages and photoacoustic images are acquired from a device that controlsthe imaging.

FIG. 3 is a diagram illustrating the configuration of imaging procedureinformation 301 generated by the examination control unit 125 based onthe examination order received from the ordering system 112. The imagingprocedure information 301 is used to acquire ultrasound images andphotoacoustic images, and generate objects for output to externaldevices at the external transmission information generating unit 127.This will be described in detail below.

The imaging procedure information 301 includes, for example, shot imageinformation 302, composite image identification information 303,superimposed image identification information 304, and related imageinformation 305. The imaging procedure information 301 may furtherinclude information indicating a imaging procedure ID uniquelyidentifying the imaging procedure information 301, imaging type, imagingsite, and imaging direction, default setting values of reconstructionparameters and imaging conditions, storage transfer settings, printoutput settings, and so forth. That is to say, the imaging procedureinformation 301 is information for implementing imaging, subjectingimage data to post-processing and transmitting to an external device,and includes information that can be set for individual imagingprocedures.

The shot image information 302 is made up of the image data imaged bythe imaging system 100, and supplemental information supplementing thisimage data. In the present embodiment, the shot image information 302and image data correspond one on one. For example, in a case where thereis a plurality of image data imaged by the same imaging procedure in anexamination, a plurality of shot image information 302 will be includedin the imaging procedure information 301.

The composite image identification information 303 includes image dataof a composite image wherein at least two or more image data out of theshot image information 302 included in the imaging procedure information301 have been composited in an overlaid state to obtain a single imagedata, and supplemental information. Composite image identificationinformation 303 of a count corresponding to the number of compositeimages created using the image data imaged by the same imaging procedureare registered in the imaging procedure information 301. Supplementalinformation of the composite image identification information 303includes an image ID uniquely identifying the composite imageidentification information 303, respective image IDs for identifyingeach of the images used to generate this composite image, superimposingmethod, and so forth.

The superimposed image identification information 304 is information foridentifying, out of the shot image information 302 included in theimaging procedure information 301, an object displayable as asuperimposed image. The superimposed image identification information304 is registered in the imaging procedure information 301 in accordanceof the number of objects displayable as superimposed images, createdusing the image data imaged by the same imaging procedures. Thesuperimposed image identification information 304 is information formanaging groups of imaged images to be used for superimposed display,for example, and includes an object ID for identifying an objectdisplayable as a superimposed image. The superimposed imageidentification information 304 may further include information of imageIDs for identifying at least two or more of image data included in thisobject.

The related image information 305 is information for handling multipleimage data that has been shot in the imaging procedure information 301as a correlated group. The related image information 305 includes agroup ID for identifying the group, and information of combinations ofat least two or more image IDs group following rules for correlationthat have been decided beforehand, for example. Examples of rules forcorrelation include correlating an image information group includingtime of irradiation of the subject body by light or ultrasound waves,and signal detection time, within a predetermined range of time.Accordingly, a photoacoustic image generated based on photoacousticwaves generated by a single irradiation by light, and an ultrasoundimage identified based on the time of irradiation by light, can begrouped.

The shot image information 302 includes, for example, failed shotinformation 306, transmission permission/non-permission information 307,image type information 308, superimposed image information 309,superimposed display history information 310, post-processing historyinformation 311, and composite image information 312. The shot imageinformation 302 further includes an image ID that uniquely identifiesthe shot image information 302, image data, image processing parameters,geometric transform parameters, placement information, information ofregion of interest, implemented imaging conditions, and implementedimaging information. The image processing parameters includereconstruction parameters regarding reconstruction of photoacousticimages, for example. Implementation imaging conditions are the actualimaging conditions at the time of implementing imaging. Implementationimaging information includes information of the time of acquiring theimage, and image ID of correlated image information, for example.Placement information is information for supplemental information forsuperimposed display on image data, including such as free annotation,marker indicating imaging direction, measurement processing, cropping,masking, and so forth.

The failed shot information 306 is information indicating whether theimage data corresponding to the shot image information 302 is aso-called failed-shot image that is not suitable for diagnosis. In acase where the image data corresponding to the shot image information302 is a failed-shot image, information indicating the reason for thefailed shot is included in the failed shot information 306.

The transmission permission/non-permission information 307 isinformation indicating whether or not to transmit image datacorresponding to the shot image information 302 to an external device.

The image type information 308 is information for identifying the typeof image data corresponding to the shot image information 302. The imagetype information 308 includes information relating to, for example,imaging method of the image data, image type, and positioning.Information of the image type is information indicating the type ofimage generated based on ultrasound wave signals, and informationindicating the type of image generated based on photoacoustic signals,for example. Examples of images generated based on ultrasound wavesignals include B-mode images and elastography images. Examples ofimages generated based on photoacoustic signals include absorptioncoefficient images and oxygen saturation images.

Information of image type may further include information indicatingwhether the shot image information 302 is a form image or a functionalimage. The image type of the image data and information of whether aform image or a functional image may be set by input of user operations,or may be set based on information set beforehand. Information relatingto positioning is information indicating whether or not the image hasbeen deformed at the time of positioning for making a composited imageor superimposed display.

The superimposed image information 309 is information relating to asuperimposed image including image data corresponding to the shot imageinformation 302.

The superimposed display history information 310 is informationindicating whether, at the time of acquiring the image datacorresponding to the shot image information 302, superimposed displaywith other image data was performed or not. In a case where superimposeddisplay was performed, the image ID of the image data that was displayedsuperimposed and information indicating the superimposing order isrecorded in the superimposed display history information 310, regardlessof the superimposing order as to the other image data in thesuperimposed display.

The post-processing history information 311 is information indicatingwhether or not superimposed display with other image data was performedat the time of performing post-processing of the image datacorresponding to the shot image information 302. In a case wheresuperimposed display was performed, the image ID of the image data thatwas displayed superimposed and the superimposing order are recorded,regardless of the superimposing order as to the other image data in thesuperimposed display.

The composite image information 312 is information indicating acomposited image in which the image data corresponding to the shot imageinformation 302 is included.

In a case where waveform data such as A-mode based on ultrasound wavesignals is generated for example, the imaging procedure information 301includes information in the image type information 308 indicating thatthis is waveform data.

The imaging procedure information 301 illustrated in FIG. 3 is oneexample. Any data structure may be used as long as capable of generatingand outputting to external devices objects that can be displayed in adisplay form suitable for the user, as described later, and not all ofthe above-described information has to be included.

FIG. 4 is a flowchart illustrating an example of processing forselecting image data to be included in a superimposed image, for outputof an object displayable as the superimposed image to an externaldevice. In the following processing, the entity carrying out theprocessing in each step is the CPU 201 or the GPU 207, unlessparticularly stated otherwise.

In step S401, the examination control unit 125 accepts instructionselecting the output form of image data. Instruction of this selectionis performed by input of user operations, or by control within thecontrol device 101. An example of an instruction of this selection beingmade by internal control of the control device 101 is a notificationindicating that irradiation by ultrasound waves and light has ended isnotified from the signal acquisition unit 123 via the imaging controlunit 122 to the examination control unit 125, thereby instructing thisselection.

In step S402, the examination control unit 125 acquires shot imageinformation corresponding to all image data acquired in the examination.For example, there are cases where, in the first examination, whilegenerating ultrasound images in real time by intermittently irradiatingthe subject by ultrasound waves, the subject is irradiated by light atan optional timing to obtain photoacoustic signals and generatephotoacoustic images. In this case, multiple photoacoustic imagesgenerated based on photoacoustic waves generated by each irradiation bylight that is performed, and shot image information corresponding toultrasound images identified based in information relating to theirradiation by light, are each acquired. Shot image informationcorresponding to a moving image made up of a series of ultrasound imagesgenerated in real time may be acquired.

In step S403, the examination control unit 125 acquires the series ofshot image information that has been correlated. The series of shotimage information that has been correlated is identified by the relatedimage information 305 included in the imaging procedure information 301.Accordingly, the series of shot image information that has beencorrelated, including a photoacoustic image generated based onphotoacoustic waves generated by a single irradiation by light, and aultrasound image identified based on information relating to theirradiation by light, is acquired.

In step S404, the external transmission information generating unit 127acquires one of image information to serve as a reference, out of theseries of shot image information acquired in step S403. The imageinformation to serve as a reference does not necessarily have to be thebase image of a superimposed image. The image information to serve as areference does not have to be included in an object for output to anexternal device. The examination control unit 125 transmits the imagingprocedure information 301 to the external transmission informationgenerating unit 127. The external transmission information generatingunit 127 acquires information of settings relating to the selection,from the settings saving unit 124, for example. For example, imageinformation corresponding to the image type that has the greatest numberof image data in the series of image data acquired in the examination istaken as the reference. The user may perform settings beforehand to seta particular image type as the reference. In another example, the imagetype that is most suitable for observation of form information in theseries of image data acquired in the examination may be taken as thereference. In the following ultrasound images generated from ultrasoundwave signals in the imaging system 100 that acquires ultrasound wavesignals and photoacoustic signals will be described as being thereference.

In step S405, the external transmission information generating unit 127selects image data to be displayed superimposed along with image datacorresponding to the reference image information. The image dataselected in step S405 does not necessarily have to be a layer image forsuperimposed display with the image data selected in step S404 as thebase image. Image data for superimposed display is selected in S405based on the image information set as the reference in step S404. Instep S405, image information other than image information not selectedin step S404 may be selected out of the correlated series of shot imageinformation that has been acquired in step S403, or further narrowingdown may be performed. In the narrowing down, image data is selectedbased on image quality and similarity of the image data, for example.The image data to be selected in step S405 is narrowed down by excludingpart of image data from multiple image data with great similarity, takenat a close time. The object for superimposed display by thelater-described processing is generated based on the combination ofimage information selected in step S404 and step S405.

In step S406, the external transmission information generating unit 127determines whether or not the combination of shot image informationselected in the processing up to step S405 is included in thesuperimposed image information 309. If included, that combination is notredundantly registered in the superimposed image identificationinformation 304, and the flow advances to step S408. If not included,the flow advances to step S407.

In step S407, the external transmission information generating unit 127adds the combination of shot image information selected in theprocessing up to step S405 in the superimposed image identificationinformation 304 of the imaging procedure information 301.

In step S408, the external transmission information generating unit 127determines whether or not determination has been completed regarding allof the series of image information that has been correlated, with regardwhether or not to take as the object of superimposed display. IFcompleted, the flow advances to step S409. If not completed, the flowadvances to step S404, and processing for this determination iscontinued.

In step S409, the external transmission information generating unit 127executes superimposed image generating processing. Details ofsuperimposed image generating processing will be described later withreference to FIG. 5.

In step S410, the external transmission information generating unit 127determines whether or not determination has been completed regarding allshot image information included in the examination with regard whetheror not to take as the object of superimposed display. If completed, theexternal transmission information generating unit 127 updates theinformation included in the imaging procedure information 301 andnotifies the examination control unit 125 that processing of thisdetermination has been completed, and the processing illustrated in FIG.4 is ended. If not completed, the flow advances to step S403, andprocessing for this determination is continued.

FIG. 5 is a flowchart illustrating an example of superimposed imagegenerating processing performed in step S409 illustrated in FIG. 4.

In step S501, the external transmission information generating unit 127acquires information relating to generating of a superimposed image fromthe settings saving unit 124. Information relating to generating of asuperimposed image is obtained by various types of settings such asexemplified in FIG. 13, for example.

In step S502, the external transmission information generating unit 127obtains one combination of image data to be the object of generating asuperimposed image or a composite image, from the superimposed imageidentification information 304 registered in the imaging procedureinformation 301.

In step S503, the external transmission information generating unit 127selects a base image for a superimposed image from the combinationobtained in step S502.

In step S504, the external transmission information generating unit 127decides a method for superimposing another image data on the base imageselected in step S503 to generate a superimposed image. The externaltransmission information generating unit 127 decides the method andorder of superimposing image data other than the image data selected instep S503 as the base image, out of the combination acquired in stepS502, upon the base image. The external transmission informationgenerating unit 127 then generates the superimposed image information309.

The method of selecting each of the base image and layer image, andsuperimposing in the processing in step S503 and step S504, is decidedbased on information in the settings 1306 illustrated in FIG. 13, forexample. In the example illustrated in FIG. 13, the method of selectingthe base image and layer image can be selected from “default selectingmethod”, “customized selecting method”, “select based on superimposeddisplay history”, and “select based on post-processing history”, insettings 1304.

In a case where “default selecting method” is set, the externaltransmission information generating unit 127 selects the base image andlayer image for the superimposed image based on the method stored in thecontrol device 101 as the default value. In a case where “customizedselecting method” is set, the external transmission informationgenerating unit 127 obtains information of the corresponding customizedsettings from the settings saving unit 124. The external transmissioninformation generating unit 127 then selects the base image and layerimage for the superimposed image based on the information of thecustomized settings.

Examples of methods that can be set by “default selecting method” and“customized selecting method” includes a selecting the base image andlayer image based on the image type information 308, for example. Forexample, in a case where image data and waveform data coexist in theimaging procedure information 301, settings can be made to generate asuperimposed image only among image data. Also, image data of aparticular type of image can be set to be taken as the base image. Forexample, a form image with detailed form information, like a B-modeimage that is an example of an ultrasound image, may be set to be takenas a base image. In a case where information indicating whether a formimage or a function image is included in the image type information 308,the base image and layer image may be selected based on this. Asuperimposed image can be efficiently interpreted by using a form imageas the base image and a function image as the layer image. In anotherexample, the combination of image data to generate the superimposedimage can be selected based on information relating to positioning inthe image type information 308. For example, in a case where a set ofimage data that has been positioned exists, a superimposed image can begenerated among the positioned image data. In a case where no sets ofimage data that have been positioned exist in the plurality of shotimage information, the external transmission information generating unit127 may control the image processing unit 121 to perform positioningprocessing.

In a case where “select based on superimposed display history” isselected, the external transmission information generating unit 127selects a base image and layer image for the superimposed image based onthe superimposed display history information 310. For example, imagedata to serve as the reference, and image data for superimposed display,are selected based on input of user operations, and the form ofsuperimposed display at the time of acquiring the image data to serve asthe reference that has been selected by previous settings as describedabove. The image type of image data included in the superimposed image,superimposing method, and superimposing order, conform to the form ofsuperimposed display at the time of acquiring the image data serving asthe reference. In a case where no information is registered in thesuperimposed display history information 310, the input/output controlunit 126 may display on the display unit 104 a screen annunciating tothe user that no image that is the object exists. A screen for the userto select a base image and layer image from the series of shot imageinformation that has been correlated, obtained in step S403, may bedisplayed.

In a case where “select based on post-processing history” is selected,the external transmission information generating unit 127 selects a baseimage and layer image for the superimposed image based on thepost-processing history information 311. For example, with regard toimage data to serve as a reference that has been selected by input ofuser operations or prior settings such as described above, for example,image data to be displayed superimposed with this image data to serve asa reference is selected based on the form of the superimposed display atthe time of having performed post-processing. The image type of imagedata included in the superimposed image, superimposing method, andsuperimposing order, conform to the form of superimposed display whenthe base image was subjected to post-processing. Specifically, at thepoint of having performed image processing or processing to change thedisplay state of this image data while making a preview display of thisimage data to serve as a reference, the external transmissioninformation generating unit 127 selects the image data displayedsuperimposed together. For example, when displaying a B-mode image thatis an example of an ultrasound image, and a region that could possiblybe a tumor and the user should perform detailed observation has beenfound, there are cases where annotation is placed at this region.Alternatively here are cases where image processing is performed to viewthis region more clearly. If a photoacoustic image or the like isdisplayed superimposed on this B-mode image at the time of performingsuch post-processing, this photoacoustic image is judged to be useful assupplementary information for interpreting. On the other hand, if notsuperimposed display is performed, judgment is made that betterinterpreting can be performed without superimposing anything.Accordingly, a form that the user has judged to be suitable forinterpreting can be reflected in the object for output to externaldevices. The external transmission information generating unit 127references the image type information 308 of each of the shot imageinformation 302 included in the superimposed image identificationinformation 304. The external transmission information generating unit127 selects image data of an image type that has detailed forminformation, out of the image data included in this combination, as thebase image. An image type that has detailed form information is, forexample, a B-mode image that is an example of an ultrasound image, asdescribed earlier. In a case where multiple images that have detailedform information, such as a B-mode image and CT image or magneticresonance imaging (MRI) image, for example, are included in thesuperimposed image identification information 304, the externaltransmission information generating unit 127 obtains the information ofexaminations of the same subject performed in the past from theexamination implementation information saving unit 120. Situations arealso conceivable where only image data having poor form information,such as some image types obtained based on photoacoustic waves, orsingle photon emission computed tomography (SPECT) images, exists in thesuperimposed image identification information 304. In this case, theexternal transmission information generating unit 127 obtainsinformation of examinations performed on the same subject in the pastfrom the examination implementation information saving unit 120. Theexternal transmission information generating unit 127 searches theexamination implementation information saving unit 120 for informationof examinations performed by the same imaging procedure. In a case whereinformation of examinations performed by the same imaging procedure isfound, the external transmission information generating unit 127 selectsa base image in the same way as the examination implemented in the past.In a case where no information of examinations performed by the sameimaging procedure is found, the external transmission informationgenerating unit 127 uses the oldest image data in the order ofacquisition as the base image, for example.

In the example in FIG. 13, the superimposing method can be selected from“superimpose individually”, “composite images for superimposing”, and“composite base image and layer image” in the settings 1306.

In a case where the superimposing method is set to “composite base imageand layer image”, in step S505 the external transmission informationgenerating unit 127 causes the image processing unit 121 to perform thiscompositing processing. The external transmission information generatingunit 127 transmits the shot image information 302 corresponding to thebase image and layer image, along with information indicating thesuperimposing method and superimposing order, to the image processingunit 121. In a case where the superimposing method is not “compositebase image and layer image”, the flow advances to step S508.

In step S506, the image processing unit 121 generates image data of thecomposited image where the base image and layer image have beensuperimposed and composited, based on the information received in stepS505. The external transmission information generating unit 127registers the image ID and image data in the composite imageidentification information 303, in order to identify the compositedimage.

In step S507, the external transmission information generating unit 127updates the information of all shot image information 302 correspondingto the image data used in the composited image generated in step S506.The external transmission information generating unit 127 registers inthe composite image identification information 303 the image ID foridentifying the composited image generated in step S506.

In a case where the superimposing method is set to “composite layerimages” in step S508, the external transmission information generatingunit 127 causes the image processing unit 121 to perform thiscompositing processing. The external transmission information generatingunit 127 transmits the shot image information 302 corresponding to baseimage and layer images, along with information indicating thesuperimposing method and superimposing order, to the image processingunit 121. In a case where the superimposing method is not “compositelayer images”, the flow advances to step S510.

In step S509, the image processing unit 121 generates a composited imagewhere multiple layer images have been composited, based on theinformation received in step S508.

In step S510, the external transmission information generating unit 127updates the information of the superimposed image information 309 in theshot image information 302 corresponding to the base image. The externaltransmission information generating unit 127 registers image IDs foridentifying the image data used in the composited image forsuperimposing on the base image in step S509, in the superimposed imageinformation 309. On the other hand, in a case where “superimposeindividually” is selected as the superimposing method, or in a casewhere the count of layer images is one, a composited image is notgenerated. In this case, an image ID corresponding to image data to besuperimposed is registered in the superimposed image information 309 ofthe shot image information 302 corresponding to the base image.

In step S511, the external transmission information generating unit 127determines whether or not the processing up to step S510 has beencompleted for all image data that is the object of generating asuperimposed image or composited image. Image data that is the object ofgenerating a superimposed image or composited image is identified fromthe superimposed image identification information 304 registered in theimaging procedure information 301. In a case where this processing isnot completed for all image data, the flow advances to step S502. Ifcompleted, the processing illustrated in FIG. 5 is ended.

The series of processing illustrated in FIG. 5 has been described asbeing an example of detailed processing of step S409 illustrated in FIG.4, but this is not restrictive. For example, the base image forsuperimposed display in step S404 in FIG. 4 may be selected, and thelayer image to superimpose on this base image layer may be selected instep S405 based on the settings illustrated in FIG. 13. The image dataselected in step S404 may then be identified in step S503, and the imagedata selected in step S405 identified in step S504, thereby realizingthe series of processing illustrated in FIG. 5.

FIGS. 6a and 6B are flowcharts illustrating an example of processing fortransmitting an object of displaying a superimposed image from thecontrol device 101 to an external device.

In step S601, the examination control unit 125 accepts an instruction tooutput information including image data to an external device.Instruction of this input may be performed by input of user operations,or may be performed having been triggered by particular processinghaving been performed. The examination control unit 125 controls theexternal transmission information generating unit 127 to generate anobject including the image data to be transmitted to the externaldevice. The object of image data to be transmitted to the externaldevice includes image data, and various types of informationsupplemented in accordance with the DICOM standard, for example. Theobject transmitted to the external device does not necessarily have toinclude image data. The examination control unit 125 transmitsinformation such as patient information, examination information, andimaging procedure information 301, that is information necessary togenerate the object, to the external transmission information generatingunit 127.

In step S602, the external transmission information generating unit 127acquires the shot image information 302 corresponding to the image dataregarding which output to the external device has been instructed instep S601.

In step S603, the external transmission information generating unit 127acquires a single image information from the shot image information 302acquired in step S602.

In step S604, the external transmission information generating unit 127obtains the failed shot information 306 included in the shot imageinformation 302. In a case where the failed shot information 306 is setto ON, the flow advances to step S605, and in a case of being set toOFF, advances to step S607.

In step S605, the external transmission information generating unit 127effects control so that the failed shot image of which the failed shotinformation 306 is ON, and a superimposed image and composited imageincluding this failed shot image, are not output to the external device.Specifically, the external transmission information generating unit 127references all image information groups correlated to this failed shotimage. In a case where the image ID of this failed shot image isregistered on the superimposed image information 309, the externaltransmission information generating unit 127 deletes the image ID ofthis failed shot image from the superimposed image information 309,composite image information 312, composite image identificationinformation 303, and superimposed image identification information 304.

In step S606, the external transmission information generating unit 127determines whether or not a storage device (omitted from illustration)for storing failed shot images is connected to the control device 101.An example of a storage device (omitted from illustration) for storingfailed shot images is an image server. Generally, failed shot images areunsuitable for interpreting, and accordingly are not transmitted to thePACS 113. However, there are cases of using failed shot images forexchanging information among users or for educational proposes, so thereare cases where this storage device is provided to manage failed shotimages. In a case where the storage device (omitted from illustration)for storing failed shot images and the control device 101 are connected,the flow advances to step S609, and in a case of not being connected,advances to step S612.

In step S607, the external transmission information generating unit 127acquires the external transmission permission/non-permission information307 included in the shot image information 302. In a case where theexternal transmission permission/non-permission information 307 is setto ON, the flow advances to step S609, and in a case of being set toOFF, advances to step S608.

In step S609, the external transmission information generating unit 127generates an image object 1102 based on the shot image informationcorresponding to the image data regarding which output to the externaldevice has been instructed in step S601.

In step S608, the external transmission information generating unit 127determines whether or not the shot image information 302 correspondingto the image data regarding which output to the external device has beeninstructed in step S601 is being used in any of a base image ofsuperimposed display, layer image, or composited image. In a case ofbeing used in any, the flow advances to step S610, and in a case ofbeing used in none, advances to step S611.

In step S610, the external transmission information generating unit 127records information of the superimposed image information 309 in theexternal transmission information 1101. The external transmissionpermission/non-permission information 307 is set to OFF in the shotimage information 302 in step S610, so individual image data is notoutput to the external device, and can be output as a superimposed imageor composite image.

In step S611, the input/output control unit 126 displays a warning onthe display unit 104. The transmission permission/non-permissioninformation 307 is set to OFF in the shot image information 302 in stepS611, and is not used in any of the base image of the superimposedimage, layer image, or composite image, so the image data correspondingto this shot image information 302 is not output to the external device.Accordingly, the input/output control unit 126 makes a popup display ofa warning dialog 1001 on the imaging screen 701 such as exemplified inFIG. 10, for example. A message for annunciating to the user that thereis image data not transmitted to an external device, and the image ID ofthe image data that is not transmitted to the external device, aredisplayed in the warning dialog 1001. Further, a continuationinstructing portion 1002 to instruct continuation of transmissionprocessing to the external device, and a cancellation instructionportion 1003 to instruct cancelling of this transmission processing aredisplayed on the warning dialog 1001. When the continuation instructionportion 1002 is pressed, the warning dialog 1001 is closed, and thetransmission processing relating to other image data is continuedwithout changing the settings of the image which are settings of notbeing transmitted to an external device. When the cancellationinstructing portion 1003 is pressed, the warning dialog 1001 is closed,and the transmission processing is canceled. In a case where settings1309 illustrated in FIG. 13 are set to OFF, the input/output controlunit 126 does not display the warning on the display unit 104.

In step S612, the external transmission information generating unit 127determines whether or not the processing up to step S611 has beencompleted for all shot image information corresponding to the image dataregarding which instruction has been given for output to the externaldevice in step S601. In a case where this is not completed, the flowadvances to step S603 and repeats the processing, and in a case of beingcompleted, the flow advances to step S613.

In step S613, the external transmission information generating unit 127acquires information relating to duplicate transmission settings, whichare setting regarding whether or not to individually transmit image dataused in a superimposed image or composite image, from the settingssaving unit 124. The duplicate transmission settings are set at settings1310 illustrated in FIG. 13, for example. In a case where the duplicatetransmission settings are ON, the flow advances to step S615, and in acase of OFF, advances to step S614.

In step S614, the external transmission information generating unit 127deletes, from shot image information registered in the externaltransmission information 1101, that corresponding to image data used ina superimposed image or composite image.

In step S615, the external transmission information generating unit 127acquires information relating to the external device regarding whichoutput has been instructed in step S601. Examples of informationrelating to the external device include information relating toDICOM-pursuant functions of the PACS 113 or viewer 114, and informationof functions relating to superimposed display. Specific examples areinformation of reception and display of Information Object Definition(IOD) objects such as grayscale softcopy presentation state (GSPS) andcolor soft copy presentation state (CSPS) in DICOM, and informationrelating to functions of performing superimposed display ofindividually-received image data. The external transmission informationgenerating unit 127 acquires information relating to the external devicefrom the PACS 113 or viewer 114 via the network 110 at an optionaltiming.

Alternatively, when connection is made between the control device 101and the PACS 113 or viewer 114, information relating to these externaldevices may be stored in the settings saving unit 124, and the externaltransmission information generating unit 127 may acquire thisinformation from the settings saving unit 124.

In step S616, the external transmission information generating unit 127generates the external transmission information 1101 for transmission tothe external device, based on the information acquired in step S615. Theexternal transmission information generating unit 127 generates theexternal transmission information 1101 based on the superimposed imageidentification information 304, composite image identificationinformation 303, and superimposed image information 309 of each shotimage information 302, of the imaging procedure information 301. Theexternal transmission information generating unit 127 decides agenerating method of the image object 1102 that is an object fordisplaying a superimposed image, based on settings relating to thetransmission method set beforehand. The settings relating to thistransmission method are set at settings 1308 illustrated in FIG. 13, forexample.

The shot image information 302 individually transmitted, and the shotimage information 302 selected as a base image of the superimposedimage, are generated as individual image information 1103 of the imageobject 1102 in the external transmission information 1101, regardless ofthe contents of settings for the transmission method in the settings1308. Also, the composite image identification information 303 isgenerated as composite image information 1104 of the image object 1102,regardless of the contents of settings for the transmission method inthe settings 1308. The superimposed image information 309 is thengenerated based on the settings set for the transmission method in thesettings 1308.

In a case where “transmit as supplementary information of base image”has been selected, the superimposed image information 309 is stored inthe superimposed image information 1108 included in the individual imageinformation 1103 of the corresponding base image. In a case where“transmit as reference object of base image” is selected, thesuperimposed image information 309 is stored as superimposing referenceimage information 1105 in the image object 1102, for each superimposedimage data. The image ID of the base image corresponding to thesupplemental information 1111 is input. In a case where “transmit asindividual image object” is selected, the superimposed image information309 is stored as individual image information 1103 in the image object1102 for each image data.

Also, the image object 1102 of the external transmission information1101 may be created conforming to a format defined in DICOM. In a caseof being created conforming to a format defined in DICOM, shot imageinformation 302 to be individually transmitted, shot image information302 selected as a base image, and composite image identificationinformation 303 are generated as a DICOM image file 1201 illustrated inFIG. 12, regardless of the transmission method settings. Thesuperimposed image information 309 is then generated based on thetransmission method settings.

In a case where “transmit as supplementary information of base image”has been selected, the superimposed image information 309 is stored inoverlay image data 1206 included in the DICOM image file 1201 of thecorresponding image (base image for superimposed display). In a casewhere “transmit as reference object of base image” has been selected,the superimposed image information 309 is generated as a GSPS object1207 for each image data. A service object pair instance uniqueidentifier (SOP Instance UID) 1204 of the corresponding base image isinput to a reference SOP Instance UID 1208. In a case where “transmit asindividual image object” has been selected, the superimposed imageinformation 309 is created as a DICOM image file 1201 for each imagedata. Note that in the following, an example of a DICOM GSPS object willbe used, but other objects may be used, such as CSPS objects or thelike.

A case is conceivable where overlay image data 1206 included in theheader information 1202 of the DICOM image file 1201 or GSPS object 1207defined in DICOM has a different bit width for pixel values from animage data group included in the imaging procedure information 301. Inthis case, the pixel values of the superimposed image information 309included in the imaging procedure information 301 are converted intovalues storable in the overlay image data 1206, and then input.

Further, the external transmission information generating unit 127 maycontrol the generating method in accordance with information relating tothe external device acquired in step S615. For example, in a case wherethe PACS 113 or viewer 114 has functions of superimposed display ofimage data individually transmitted, a generating method the same as the“transmit as individual image object” settings is used. Accordingly, theuser who has displayed this image object using the viewer 114 candisplay as a superimposed image for example, and also can displayindividual images. Also, in a case of the external device havingfunctions of receiving and displaying a GSPS object, a generating methodthe same as the “transmit as reference object of base image” settings isused. In a case where the external device has neither of thesefunctions, a generating method the same as the “transmit assupplementary information of base image” settings is used.

In step S617, the transmission/reception control unit 128 transmits theobject generated in step S616 to the external device. The externaltransmission information generating unit 127 transmits the generatedexternal transmission information 1101 to the examination control unit125. The examination control unit 125 transmits the externaltransmission information 1101 to the external device via thetransmission/reception control unit 128.

FIG. 7 is a diagram illustrating an example of a imaging screen 701displayed on the display unit 104 by the control device 101. The imagingscreen 701 includes a related image list 702, a related image item 703,a superimposing instruction portion 704, a failed-shot instructingportion 705, a transmission instructing portion 706, an image displayportion 707, a superimposed image switching portion 708, a superimposedimage adding portion 709, a superimposed image finalizing portion 710, asuperimposed image deleting portion 711, a composite image switchingportion 712, a composite image display instructing portion 713, acomposite image adding portion 714, a composite image finalizing portion715, a composite image deleting portion 716, an annotation input portion717, an annotation placement portion 718, an annotation deleting portion719, a transmissive display instruction portion 720, a mixed displayinstruction portion 721, a superimposed display instruction portion 722,a list confirmation portion 723, a thumbnail image display portion 724,thumbnail images 725, and page switching instruction portions 726.

The related image list 702 is a list of all images correlated with athumbnail images 725 a that is in a selected state in the thumbnailimage display portion 724. In a case where all information of therelated image list 702 cannot be displayed on the display unit 104 as alist, the display of the related image list 702 is switched byscrolling. The related image item 703 is displayed in each row of therelated image list 702. Although the related image list 702 is shown asbeing distinguished according to imaging types in the example in FIG. 7,images taken in a correlated manner in the examination and correspondingrelated image items 703 may be displayed included in a single list.

The related image item 703 is a list item corresponding to image data ina one on one manner. A text string representing the name of the imagedata, the superimposing instruction portion 704, the failed-shotinstructing portion 705, and the transmission instructing portion 706are displayed within the related image item 703. A thumbnail image ofthe corresponding image data may be displayed in the related image item703 instead of the text string representing the name of the image data,or both the text string and thumbnail image may be displayed together.The user can perform input operations regarding the related image item703, and can select the related image item 703. Image data correspondingto the selected related image item 703 is displayed on the image displayportion 707. The related image item 703 corresponding to the B-modeimage of ultrasound images is selected in the example illustrated inFIG. 7, and a B-mode image 727 is displayed in the image display portion707.

The superimposing instruction portion 704 is a button that instructs forthe image corresponding to the related image item 703 to be superimposedon the image preview-displayed in the image display portion 707. Whenthe superimposing instruction portion 704 is set to ON, thecorresponding image is displayed superimposed as a layer image on thepreview-displayed image. When the superimposing instruction portion 704is set to OFF, the image that was displayed superimposed on thepreview-displayed image is hidden from display. In the example in FIG.7, an oxygen saturation image 728 and an oxyhemoglobin amount 729 aredisplayed superimposed on the preview-displayed B-mode image 727. Notethat the superimposing instruction portion 704 will be set to OFF forrelated image items 703 regarding which superimposed display would bemeaningless. For example, A-mode waveform data and M-mode temporal axisdata generated from ultrasound wave signals have the superimposeddisplay regarding B-mode images set to OFF.

The failed-shot instructing portion 705 is a button for performingfailed-shot processing regarding image data corresponding to the relatedimage item 703. When failed-shot processing is executed, the failed shotinformation 306 corresponding to this image data is switched to ON. Afailed-shot mark indicating that this is a failed-shot image is alsodisplayed on the related image item 703. When the failed-shot executionis cancelled, the failed shot information 306 is switched to OFF. Thefailed-shot mark in the related image item 703 also is hidden fromdisplay.

The transmission instructing portion 706 is a button for individuallyinstructing whether or not to transmit the image data corresponding tothe related image item 703 to an external device. When the transmissioninstructing portion 706 is switched to ON, the transmissionpermission/non-permission information 307 corresponding to this imagedata is switched to ON. The transmission instructing portion 706 isdisplayed in different forms according to the ON and OFF settings. Inthe example in FIG. 7, in the case of ON, display is made in the form ofa transmission instructing portion 706 a, and in the case of OFF,display is made in the form of a transmission instructing portion 706 b.

Thus, failed-shot processing and transmission to external devices can beset for each related image item 703, i.e., for each image data, at thecontrol device 101. The failed-shot instructing portion 705 andtransmission instructing portion 706 can also be set separately, so thecontrol device 101 can distinguish whether the reason of not beingtransmitted to an external device is due to a failed shot or not. Notethat in the example in FIG. 7, failed-shot instructing portions 705 andtransmission instructing portions 706 are displayed regarding allrelated image items 703 in the related image list 702, and are settable.This example is not restrictive, however, and an arrangement may be madewhere the failed-shot instructing portion 705 and transmissioninstructing portion 706 are displayed and selectable only for therelated image item 703 corresponding to image data displayed in theimage display portion 707.

The image display portion 707 is a region for displaying medical imagesobtained by the imaging system 100. Images displayed in the imagedisplay portion 707 may be any type of image, such as still images,moving images, waveform data, and so forth. An image ispreview-displayed in the image display portion 707 in real-time whileimaging a moving image. Image data corresponding to the related imageitem 703 selected from the related image list 702 is preview-displayedon the image display portion 707.

The superimposed image switching portion 708 is a region for switchingsuperimposed display registered in the superimposed image information309 corresponding to the image data preview-displayed in the imagedisplay portion 707. The superimposed image switching portion 708displays options of “no display”, and selection from superimposeddisplays registered in the superimposed image information 309. Imagedata selected at the superimposed image switching portion 708 isdisplayed as a layer image for the image data preview-displayed in theimage display portion 707. In a case where “no display” is selected, noimage data is superimposed.

The superimposed image adding portion 709 is a button for adding thestate superimposed-displayed on the image display portion 707, to thesuperimposed image information 309 of image data corresponding to therelated image item 703 selected so as to make the preview display in theimage display portion 707. Superimposed image information 309 reflectingthe form of the superimposed display in the image display portion 707 atthe point that the superimposed image adding portion 709 is pressed isgenerated. This form of superimposed display is a form regarding imagedata superimposed as layer images, and the superimposing order thereof.If there has been no registration made reflecting the same superimposeddisplay form of the superimposed image information 309 generated, in thesuperimposed image information 309 of image data corresponding to therelated image item 703 selected for the preview display in the imagedisplay portion 707, this is newly added. When information of thesuperimposed image information 309 is updated, the options of thesuperimposed image switching portion 708 also are updated.

The superimposed image finalizing portion 710 is a button for confirmingthe superimposed display form, and instructing updating of the contentsof the superimposed image information 309. The superimposed display formselected at the superimposed image switching portion 708 is updated tocontents reflecting the superimposed display form at the point that thesuperimposed image finalizing portion 710 was pressed. In the exampleillustrated in FIG. 7, the superimposed display form registered as“superimposed image 1” is updated to the form at the point at which thesuperimposed image finalizing portion 710 is pressed. Accordingly, thesuperimposed image information 309 is finalized. In a case where “nodisplay” is selected from the superimposed image switching portion 708,the superimposed image finalizing portion 710 is disabled.

The superimposed image deleting portion 711 is a button for instructiondeletion of the contents of the superimposed image information 309 thathave been registered. When the superimposed image deleting portion 711is pressed, information corresponding to the selected superimposeddisplay form is deleted from the superimposed image information 309, anddeleted from the options in the superimposed image switching portion708. Further, superimposed display in the image display portion 707 iscancelled. In a case where “no display” has been selected from thesuperimposed image switching portion 708, the superimposed imagedeleting portion 711 is disabled.

The composite image switching portion 712 is a region for switchingdisplay of composite image data corresponding to the composite imageidentification information 303 registered in the imaging procedureinformation 301. The composite image switching portion 712 displaysoptions of “no display”, and selection from composite images registeredin the imaging procedure information 301.

The composite image instructing portion 713 is a button for instructingdisplay of the composite image selected at the composite image switchingportion 712 in the image display portion 707. At the point that thecomposite image instructing portion 713 is pressed, the composite imageselected at the composite image switching portion 712 is displayed inthe image display portion 707. In a case where “no display” is selectedat the composite image switching portion 712, no composite image isdisplayed in the image display portion 707. At the point that thecomposite image display instructing portion 713 is pressed, image datadisplayed at the image display portion 707 is hidden from display.

The composite image adding portion 714 is a button for instructinggenerating of a composite image where a plurality of image datasuperimposed-displayed in the image display portion 707 has beencomposited, and addition to the composite image identificationinformation 303 in the imaging procedure information 301. If compositeimage identification information 303 of the same form as thatsuperimposed-displayed in the image display portion 707 has not beenregistered in the imaging procedure information 301, this is newlyadded. When the information of the composite image identificationinformation 303 is updated, the options in the composite image switchingportion 712 are also updated.

The composite image finalizing portion 715 is a button for instructionto finalize the superimposed display form of the composite image, andupdate the contents of the composite image identification information303. The superimposed display form of the composite image selected atthe composite image switching portion 712 is changed to the contentsreflecting the superimposed display form at the point that the compositeimage finalizing portion 715 was pressed. In the example illustrated inFIG. 7, the superimposed display form registered as “composite image 2”is updated to the superimposed display form at the point that thecomposite image finalizing portion 715 was pressed. Thus, the compositeimage identification information 303 is updated. In a case where “nodisplay” has been selected at the composite image switching portion 712,the composite image finalizing portion 715 is disabled.

The composite image deleting portion 716 is a button for instructingdeletion of the registered contents of the composite imageidentification information 303. When the composite image deletingportion 716 is pressed, the information corresponding to the selectedcomposite image is deleted from the composite image identificationinformation 303, and deleted from the options in the composite imageswitching portion 712. Display of the composite image in the imagedisplay portion 707 is cancelled. In a case where “no display” isselected at the composite image switching portion 712, the compositeimage deleting portion 716 is disabled.

The annotation unit portion 717 is a region for inputting a text stringfor annotation to be placed in the image display portion 707. Theannotation placement portion 718 is a button for instructing placementof the text string input to the annotation unit portion 717 on the imagedata displayed in the image display portion 707. The text string inputto the annotation input portion 717 at the point that the annotationplacement portion 718 is pressed becomes annotation data, and is placedin the image display portion 707. The annotation deleting portion 719 isa button for instructing deletion of an annotation placed in the imagedisplay portion 707. Upon pressing the annotation deleting portion 719in a state where one of the annotations placed in the image displayportion 707 is selected, the selected annotation is deleted.

The transmissive display instruction portion 720 is a region forchanging the transmissivity of the layer image displayed superimposed inthe image display portion 707. The mixed display instruction portion 721is a button for instructing switching of the superimposed display in theimage display portion 707 to mixed display. This mixed display is adisplay method where pixel values of overlaid pixels are mixed anddisplayed in the superimposed display of a plurality of image data. Themixed display corresponds to an example of “display overlaid portionsusing different color” in illustrated in FIG. 8, for example. Thesuperimposed display instruction portion 722 is a button for instructingto specify the superimposing order of layer images for superimposeddisplay in the image display portion 707, and switch to superimposeddisplay according to the specified order. Note that one is selected fromthe three of the transmissive display instruction portion 720, mixeddisplay instruction portion 721, and superimposed display instructionportion 722, and the superimposed display method in the image displayportion 707 switches in conjunction with switching of which of these isspecified.

The list confirmation portion 723 is a button for instructing displayinga list of external transmission information 1101 on the display unit 104at the point of having been pressed. When the list confirmation portion723 is pressed, a list screen 901 pops up on the imaging screen 701.

The thumbnail image display portion 724 is a region displaying thumbnailimages corresponding to image data selected as references from theplurality of image data correlated in the examination. The thumbnailimage display portion 724 includes the failed-shot instructing portion705, transmission instructing portion 706, thumbnail images 725, andpage switching instruction portions 726. The example in FIG. 7illustrates display of thumbnail images of a series of B-mode imagesthat are an example of ultrasound images, in the time sequence ofacquisition. For example, there are conceivably cases where, whiletaking ultrasound images, photoacoustic signals are acquired at anoptional timing to obtain photoacoustic signals, and photoacousticimages are acquired. It is further conceivable that the acquiredultrasound images and photoacoustic images will be compared and observedby superimposed display, for example. As described above, the controldevice 101 selects an ultrasound image as a reference image, forexample. Accordingly, the ultrasound image selected as a reference isdisplayed in the thumbnail image display portion 724, and an image groupcorrelated with the selected ultrasound image is displayed in therelated image list 702. Note that the image type selected as a referenceis not restricted to ultrasound images, and can be set and selectedoptionally, as described above.

For each bunch of image data acquired in one imaging procedure, forexample, a thumbnail image 725 corresponding to each image data isdisplayed at the thumbnail image display portion 724. In a case where amoving image is acquired, a thumbnail image 725 corresponding to arepresentative frame image out of the multiple frames included in themoving image is displayed. Thumbnail images 725 corresponding to thesequentially-acquired image data are added to the thumbnail imagedisplay portion 724 in the imaging screen 701, each time image data isacquired, until the examination ends.

In response to the input of operation regarding a thumbnail image 725displayed in the thumbnail image display portion 724, image data to bepreview-displayed in the image display portion 707 is selected. Theselected thumbnail image 725 goes to a preview-selected state (725 a),and thumbnail images 725 that are not selected are in anon-preview-selected state (725 b). Also, in conjunction with thethumbnail image 725 preview-displayed in the image display portion 707being switched, the display of the failed-shot instructing portion 705and transmission instructing portion 706 included in the thumbnail imagedisplay portion 724, and the related image item 703 displayed in therelated image list 702, are updated. The failed-shot instructing portion705 and transmission instructing portion 706 included in the thumbnailimage display portion 724 indicate the failed-shot settings andtransmission permission/non-permission settings regarding the thumbnailimage 725 a in the preview-selected state. In a case where all thumbnailimages 725 cannot be displayed in the thumbnail image display portion724, the page switching instruction portions 726 displayed are enabled.Pressing the page switching instruction portions 726 at the left andright of the thumbnail image 725 group switches the list of thumbnailimages 725 displayed in the thumbnail image display portion 724.

FIGS. 8A through 8H are diagrams illustrating an example of superimposeddisplays set by the control device 101 and applied. The superimposingmethod of data for superimposed display using a plurality of image data,or composited image data, is switched by settings thereof. Thissuperimposing method is selected from “transmissive display”, “displayoverlaid portions using different color”, “superimposed displayaccording to order”, “masked display” and “difference display”, forexample. Note that it is sufficient for the superimposing method to bestipulated for image processing using a plurality of image data, and isnot restricted to the above options. An example of a case of using twoimage data of FIGS. 8A and 8B will be described below.

FIG. 8C is an example of display in a case where the superimposingmethod is set to “transmissive display”. The user can set thetransmissivity of the layer image. At least one of the two image data isdisplayed at this set transmissivity.

FIG. 8D is an example of display in a case where the superimposingmethod is set to “display overlaid portions using different color”. Thisis a display by a different color from that in a case where at least oneof the two image data is displayed independently. Accordingly, the usercan more readily visually recognize overlaid regions when superimposingtwo image data. For example, pixel values are decided for pixel valuesin the two image data having the same coordinates, and a superimposedimage is displayed with a color where the display colors of each havebeen mixed. Where a pixel value is set for only one image data, andthere are coordinates in the other image data where no pixel valuesexists or pixel values are 0, display may be made using the pixel valuesof the former image data, i.e., display using the original color of theformer image data.

FIGS. 8E and 8F are examples of a case of displaying with thesuperimposing method set to “superimposed display according to order”.The user can set the superimposing order of layer images. FIG. 8E is acase where the user has set the image data in FIG. 8A as the layerimage, and FIG. 8F is a case where the user has set the image data inFIG. 8B as the layer image.

FIG. 8G is an example of a display where the superimposing method hasbeen set to “masked display”. In the masked display, only coordinateswhere pixel values are overlaid among the base image and superimposedmultiple layer images are displayed. Only pixel values for the baseimage are displayed for coordinates where pixel values exist in thelayer image, and coordinates where there are no pixel values in thelayer image are masked.

FIG. 8H is an example of display in a case where the superimposingmethod is set to “difference display”. In the difference display,coordinates where pixel values exist only in one of the base image andthe superimposed layer images are displayed. Coordinates where pixelvalues exist in both base image and layer images are masked. Coordinateswhere pixel values exist in only one are displayed in the same way asthe superimposing method “superimposed display according to order”.

Thus, enabling switching among multiple superimposed display methods todisplay allows superimposed display corresponding to variousinterpreting usage cases to be easily performed.

FIG. 9 is a diagram illustrating an example of a screen for displayingexternal transmission information 1101 including objects forsuperimposed display to be transmitted to an external device, as a list.The list screen 901 is a screen where external transmission information1101 can be confirmed in a list, for each imaging procedure in theexamination being performed. The list screen 901 includes a list displayswitching portion 902, a imaging procedure name 903, a list displayportion 904, and an end instruction portion 905.

The list display switching portion 902 is a region for switching betweencontent to display as a list in the list display portion 904. In theexample illustrated in FIG. 9, the content to be displayed in a list canbe switched between “superimposed image” and “composite image”. In acase where all external transmission information 1101 to be displayedcannot be displayed in the list display portion 904, a button forswitching the groups of information displayed in the list displayportion 904 will be displayed.

The imaging procedure name 903 is a region where the name of the imagingprocedure relating to the image data displayed in the list displayportion 904 is displayed.

The list display portion 904 is a region for displaying a list ofexternal transmission information 1101. The external transmissioninformation 1101 at the point that input of operation has been made atthe list confirmation portion 723 illustrated in FIG. 7 and the listscreen 901 is displayed, is displayed in the list display portion 904.In a case where “superimposed image” is selected as content to display alist of, the base image and layer images to be superimposed on this baseimage, are displayed in the list display portion 904. Image data to beindividually transmitted is also displayed in the column of base imagesin the list display portion 904. One layer image is displayed in onecell in the list display portion 904. A state where the layer image hasbeen superimposed on the base image following the superimposing methodset beforehand is displayed in one cell. On the other hand, in a casewhere “composite image” is selected as the content to be displayed as alist, composited images where base images and layer images have beencomposited following the compositing method decided beforehand, aredisplaced in the list display portion 904. Although thumbnail images aredisplayed in the example illustrated in FIG. 9, this may be expressed byany form as long as the image type and superimposing method can becomprehended.

The end instruction portion 905 is a button for ending the list displayof external transmission information 1101. When the end instructionportion 905 is pressed, the list screen 901 is closed.

FIG. 11 is a diagram illustrating an example of the detailed structureof the external transmission information 1101 including objectstransmitted to external devices. The external transmission information1101 is information for transmitting image data acquired in examinationsperformed by the imaging system 100, image data for superimposed displaythat are generated from imaged image data, and image objects 1102including composite images, together to external devices. Note however,that the control device 101 does not have to transmit all image objects1102 at once, and may transmit over several times.

The image object 1102 includes at least one image information. The imageinformation included in the image object 1102 is, for example,individual image information 1103, composite image information 1104, orsuperimposing reference image information 1105. A plurality of any ofthese image information may be included in the image object 1102, and ifthe relevant corresponding image data is not transmitted, thecorresponding image information does not have to be included.

The individual image information 1103 includes image data imaged by theimaging system 100, and related information. The individual imageinformation 1103 includes, for example, supplementary information 1106and image data 1107. The supplementary information 1106 includes patientinformation, examination information, image ID, and like information.The supplementary information 1106 may include any data that identifiesimage data included in the image data 1107 and superimposed imageinformation 1108. The supplementary information 1106 is recorded asheader information of the image data 1107, for example. Thesupplementary information 1106 may be in any form, as long as a formatthat can be read by the PACS 113 or viewer 114. A plurality ofsuperimposed image information 1108 may be included in the individualimage information 1103. In a case where a plurality of superimposedimage information 1108 is included, the individual image information1103 includes, in the supplementary information 1106 for example,information whereby the superimposing order of each image data includedin the superimposed image information 1108 can be identified. Theindividual image information 1103 correlates the superimposed imageinformation 1108 including image data of layer images to be superimposedon image data 1107 serving as the base image. The individual imageinformation 1103 is an example of an object that can be displayedsuperimposed at the external device.

The composite image information 1104 includes image data of compositeimages obtained by compositing a plurality of image data and relatedinformation. The composite image information 1104 includes, for example,supplementary information 1109 and image data 1110. The supplementaryinformation 1109 includes information such as patient information,examination information, and image ID. The supplementary information1109 may include any data that identifies the image data 1110. Thesupplementary information 1106 is recorded as the header information ofthe image data 1110, for example. The supplementary information 1109 maybe in any form, as long as a format that can be read by the PACS 113 orviewer 114. The supplementary information 1109 further includes imageIDs of individual image information 1103 corresponding to image dataused to generate the composite image, and information indicating thecompositing method. The image data 1110 is image data of this compositeimage.

The superimposing reference image information 1105 includes informationfor referencing the base image, and image data of layer images. Thesuperimposing reference image information 1105 includes, for example,supplementary information 1111 and image data 1112. The supplementaryinformation 1111 includes information such as patient information,examination information, and image ID. The supplementary information1111 may include any data that identifies the image data 1112. Thesupplementary information 1111 is recorded as header information of theimage data 1112, for example. The supplementary information 1111 may bein any form, as long as a format that can be read by the PACS 113 orviewer 114. The supplementary information 1111 also includes thesuperimposing method for superimposed display, and information of theimage ID of the individual image information 1103 corresponding to thebase image. The image data 1112 is image data of layer images. Aplurality of superimposing reference image information 1105 mayreference this individual image information 1103 as a base image. Inthis case, information by which the superimposing order of the multiplelayer images can be identified is included in at least one of thesupplementary information 1111 of the superimposing reference imageinformation 1105, or the supplementary information 1106 of theindividual image information 1103 of the base image. The superimposingreference image information 1105 correlates the layer images to besuperimposed as to the image data referenced as the base image. Thesuperimposing reference image information 1105 is an example of anobject where superimposed display can be performed at an externaldevice.

FIGS. 12A and 12B are diagrams illustrating an example of the structureof generating an object to be transmitted to an external devicefollowing the DICOM standard. An object to be generated following theDICOM standard includes information of at least one of examinationinstance ID, series UID, patient ID, examination UID, and information ofdate and time relating to examination. The information of date and timerelating to examination includes information of at least one ofexamination date, examination time, time of acquisition of ultrasoundimage, and time of acquisition of photoacoustic image. The time ofacquisition of ultrasound image is, for example, the time at whichacquisition of all ultrasound wave signals has been completed togenerate one B-mode image. The time of acquisition of photoacousticimage is, for example, the time at which photoacoustic signals used forgenerating this photoacoustic image were obtained. The object in thepresent embodiment is transmitted by two types of methods set by thesettings 1308 in a superimposed image settings screen 1301 exemplarilyillustrated in FIG. 13.

FIG. 12A is a diagram illustrating an example of the configuration where“transmit as supplementary information of base image” or “transmit asindividual image object” is selected in the settings 1308. A DICOM imagefile 1201 that is an object to be transmitted to an external deviceincludes header information 1202 and image data 1203. The headerinformation 1202 includes at least a SOP Instance UID 1204 that uniquelyidentifies the DICOM image file 1201, text string data 1205 of patientinformation, examination information, and so forth, and overlay imagedata 1206. The overlay image data 1206 is image data of a layer imagesuperimposed on the image data 1203. In a case where the settings 1308are set to “transmit as supplementary information of base image”,multiple overlay image data 1206 may be registered to one DICOM imagefile 1201. In a case where the settings 1308 are set to “transmit asindividual image object”, the overlay image data 1206 is not generated.The DICOM image file 1201 exemplified in FIG. 12A corresponds to theindividual image information 1103 in FIG. 11, for example. Note that acomposite image is generated as a single DICOM image file 1201. TheDICOM image file 1201 including the composite image corresponds to thecomposite image information 1104 illustrated in FIG. 11, for example.

FIG. 12B is a diagram illustrating an example of the configuration in acase where “transmit as reference object of base image” is set for thesettings 1308. The DICOM image file 1201 includes the header information1202 and the image data 1203. The header information 1202 includes SOPInstance UID 1204, and text string data 1205 of patient information,examination information, and so forth. The header information 1202 doesnot include overlay image data 1206. Overlay image data corresponding tothe layer image superimposed on the image data 1203 is generatedindividually as a GSPS object 1207, for example. The GSPS object 1207includes the base image on which the overlay image data 1206 is to beoverlaid, and the reference SOP Instance UID 1208 to identify thecorresponding DICOM image file 1201. The GSPS object 1207 includes textstring data 12056 such as patient information, examination information,and so forth, and overlay image data 1206. The layer image data and GSPSobject 1207 are generated in a one on one manner. Multiple GSPS objects1207 can set one DICOM image file 1201 to the reference SOP Instance UID1208. The GSPS object 1207 exemplified in FIG. 12B corresponds to thesuperimposing reference image information 1105 illustrated in FIG. 11,for example. According to this configuration, the control device 101 cantransmit objects for superimposed display at external devices.

FIG. 13 is a diagram illustrating an example of a screen to performsettings regarding superimposed display. The superimposed image settingsscreen 1301 in FIG. 13 performs settings relating to superimposeddisplay for each imaging procedure. The superimposed image settingsscreen 1301 includes imaging procedure name 1302, settings 1303 through1310, a cancel portion 1312, and an OK portion 1313.

The imaging procedure name 1302 is a region for displaying the name ofthe imaging procedure as a settable increment in the superimposed imagesettings screen 1301. In the settings 1303, whether to apply settings toall image data acquired in the imaging system 100 or not, i.e., whetheror not to perform batch settings, is selected. If not batch settings,superimposed display is performed based on settings set for each imagingprocedure.

Although an example of performing settings related to superimposeddisplay for each imaging procedure is illustrated in FIG. 13, this isnot restrictive, and settings may be made in any other increment. Forexample, settings relating to superimposed display may be applied to allimage data acquired in the imaging system 100 by batch settings. Forexample, in the case of batch settings, the imaging procedure name 1302and settings 1303 are hidden from display, and the superimposed imagesettings screen 1301 is displayed with the display region moved up.

A method for selecting image data to be included in the superimposeddisplay for output, to an external device, of an object displayable as asuperimposed image is selected at the settings 1304. Selection is madein the settings 1304 from “default selecting method”, “customizedselecting method”, “select based on superimposed display history”, and“select based on post-processing history”.

Customization settings relating to superimposed display are performed atthe settings 1305. In FIG. 13, the settings 1305 is a button forinstructing displaying of a customization settings screen 1401.Customization settings will be described later with reference to FIG.14.

The method of superimposing image data other than the image dataselected as the base image, on this base image, is selected at thesettings 1306. Selection is made from, for example, “superimposeindividually”, “composite layer images”, and “composite base image andlayer image” at the settings 1306.

The display method of the superimposed display is selected at thesettings 1307. Selection is made from, for example, “transmissivedisplay”, “display overlaid portions using different color”,“superimposed display according to order”, “masked display” and“difference display” at the settings 1307. Examples of superimposeddisplay performed by the respective display methods are illustrated inFIGS. 8A through 8H. Note that the settings 1307 are also used whengenerating composite images as well.

The method of transmitting an object for superimposed display isselected at the settings 1308. Selection is made from, for example,“transmit as supplementary information of base image”, “transmit asreference object of base image”, and “transmit as individual imageobject” at the settings 1308.

At the settings 1309, whether or not to display the warning dialog 1001in a case where there is image data that is not used for a base image orlayer image for superimposed display, not used in a composite image, andnot individually transmitted either, at the time of instructing externaltransmission, is selected. In a case where the settings 1309 are ON, thewarning dialog 1001 is displayed in a case where there is image data nottransmitted to an external device. In a case where the settings 1309 areOFF, the warning dialog 1001 is not displayed even if there is imagedata that is not transmitted to an external device.

Whether or not to individually transmit image data used in asuperimposed image or composite image is selected at the settings 1310.In a case where the settings 1310 are ON, an individual image objectincluding image data regarding which the external transmissionpermission/non-permission information 307 is set to ON is transmitted toan external device, regardless whether or not this image data is used asa base image or layer image for superimposed display or in a compositeimage. In a case where the settings 1310 are OFF, the individual imageobject including image data used as a base image or layer image forsuperimposed display or in a composite image is not transmitted to anexternal device regardless of the external transmissionpermission/non-permission information 307. Accordingly, the amount ofdata transmitted to external devices can be reduced.

The cancel portion 1312 is a button for instructing discarding of thecontents set in the superimposed image settings screen 1301. Pressingthe cancel portion 1312 closes the superimposed image settings screen1301.

The OK portion 1313 is a button for instructing finalization of thecontents set in the superimposed image settings screen 1301. Pressingthe OK portion 1313 saves the contents that have been set in thesettings saving unit 124, and the superimposed image settings screen1301 is closed.

FIG. 14A is a diagram illustrating an example of a customizationsettings screen 1401. The customization settings screen 1401 is a screenfor performing customization settings relating to superimposed display.The customization settings screen 1401 includes a list display switchingportion 1402, an adding portion 1403, an editing portion 1404, a imagingprocedure name 1405, a list display portion 1406, a cancel portion 1407,and an OK portion 1408.

The list display switching portion 1402 is a region for instructingswitching of contents displayed as a list in the list display portion1406. In the example illustrated in FIG. 14A, selection of the contentto be displayed as a list is made from “superimposed image” and“composite image”. In a case where the display content does not all fitin the list display portion 1406, buttons for switching the contentdisplayed in the list display portion 1406 are displayed in the listdisplay switching portion 1402.

The adding portion 1403 is a button for instruction of adding newcustomization settings. When the adding portion 1403 is pressed, a popupdisplay is made of an individual settings dialog 1409 exemplarilyillustrated in FIG. 14B on the customization settings screen 1401. Theuser can add customization settings by inputting into the individualsettings dialog 1409.

The editing portion 1404 is a button for instructing editing of thecustomization settings selected at the list display portion 1406. Whenthe editing portion 1404 is pressed, a popup display is made of theindividual settings dialog 1409 on the customization settings screen1401. The user can edit selected customization settings by inputtinginto the individual settings dialog 1409.

The imaging procedure name 1405 is a region for displaying the name ofthe imaging procedure as a settable increment in the customizationsettings screen 1401.

The list display portion 1406 is a region displaying the contents ofcustomization settings relating to superimposed display in a list. In acase where “superimposed image” is selected as the contents to displayin the customization settings screen 1401, information indicating a baseimage, and layer images to be superimposed on the base image, aredisplayed in the list display portion 1406. Information relating to oneimage data is displayed in one cell in the list display portion 1406.The type of image of the image data, and superimposing method as to thebase image, are displayed in individual cells. In a case where one layerimage is a composite image obtained by compositing multiple image data,the image types of all image data used in this composite image aredisplayed in the cell. In a case where “composite image” is selected asthe content to display in the customization settings screen 1401, imagedata to use as the base image when compositing, and layer images to besuperimposed on this base image and composited, are displayed in thelist display portion 1406. The image types of the image data of thelayer images, and the compositing method as to the base image, isdisplayed. Although an example where information indicating the imagetype and superimposing method is displayed as text strings in the cellsin the list display portion 1406 is illustrated in FIG. 14A, other formssuch as thumbnail images or the like may be displayed, as long as a formthat the user can understand this information.

The cancel portion 1407 is a button for instructing discarding of theediting contents of the customization settings. When the cancel portion1407 is pressed, the contents of the edited customization settings arediscarded, and the customization settings screen 1401 is closed.

The OK portion 1408 is a button for instructing finalization of theediting contents of the customization settings. When the OK portion 1408is pressed, the contents of the edited customization settings arefinalized, and information of these settings is saved in the settingssaving unit 124. The customization settings screen 1401 is then closed.

FIG. 14B is an example of the individual settings dialog 1409 forindividually performing customization settings relating to superimposeddisplay. The customization settings screen 1401 includes a imagingprocedure name 1410, settings 1411 through 1413, a cancel portion 1414,and an OK portion 1415.

The imaging procedure name 1410 is a region for displaying the name ofthe imaging procedure as a settable increment in the customizationsettings screen 1401.

The image type of the image data used for the base image is selected atthe settings 1411. The image type of the image data to be displayedsuperimposed on the base image, or of image data to be composited, i.e.,layer image, is selected at the settings 1412. Images of image typesother than the image type of the base image can each be selected aslayer images, and the order of superimposing can be set at the settings1412. A superimposing method selecting portion 1413 is the same as thesettings 1307 exemplarily illustrated in FIG. 13, so the abovedescription will be incorporated here by reference, and detaileddescription will be omitted.

The cancel portion 1414 is a button for instructing discarding of theediting contents of the individual settings. When the cancel portion1414 is pressed, the contents of the edited individual settings arediscarded, and the individual settings dialog 1409 is closed. The OKportion 1415 is a button for instructing finalization of the editingcontents of the individual settings. When the OK portion 1415 ispressed, the contents of the edited individual settings are finalized,the individual settings dialog 1409 is closed. The display contents ofthe list display portion 1406 in the customization settings screen 1401are edited by way of the individual settings dialog 1409, and finalizedcontents are updated.

Thus, the plurality of image data obtained by the imaging system 100 isoutput as objects suitable for superimposed display at the PACS 113 orviewer 114, for example. An example of acquiring ultrasound wave signalsand photoacoustic signals by the imaging system 100 and outputting imagedata generated from the acquired signals to the external device will bedescribed.

The signal acquisition unit 123 acquires ultrasound wave signals andphotoacoustic signals under control of the imaging control unit 122 andexamination control unit 125, based on information obtained from theordering system 112 or information set by the user. For example, theprobe 102 is brought into contact with the subject body by the user, thesubject body is irradiated by ultrasound waves, reflected echoes arereceived, and a series of ultrasound wave signals is obtained. A seriesof ultrasound images is displayed on the display unit 104 by the imageprocessing unit 121 and input/output control unit 126.

The user performs input of operations as appropriate to irradiate thesubject body by light from the probe 102 at a desired timing, receivesacoustic waves, and obtains photoacoustic signals. An absorptioncoefficient image that is an example of a photoacoustic image isdisplayed on the display unit 104 superimposed on the ultrasound image,by the image processing unit 121 and input/output control unit 126.

Irradiation of the subject body by ultrasound waves and light ends, andthe control device 101 starts processing for selecting the output formof the acquired image data (FIG. 4). The examination control unit 125acquires shot image information 302 (FIG. 3) relating to the series ofultrasound images, and the photoacoustic image acquired in conjunctionat a desired timing (step S402). The examination control unit 125 thenidentifies the photoacoustic image acquired at the desired timing, andultrasound images acquired at a time near this timing, as a correlatedseries of images, and acquires shot image information 302 thereof (stepS403), for example.

Combinations of image data for generating objects for superimposeddisplay are identified based on settings exemplarily illustrated in FIG.13 (steps S404 through S408). The image data to be used for superimposeddisplay is selected, and the superimposing method is decided (FIG. 5).The external transmission information generating unit 127 selects, forexample, a B-mode image that is an example of an ultrasound image as thebase image, and an oxygen saturation image that is an example of aphotoacoustic image as a layer image, based on the settings exemplarilyillustrated in FIGS. 13 through 14B (steps S502 through S503). Theseselections are performed based on customization settings made by theuser, for example. A desired form is selected from various superimposeddisplay forms such as illustrated in FIGS. 8A through 8H, based on usersettings. A composite image is then generated (steps S506 through S510),or data for superimposed display is generated (step S510).

An object for superimposed display is transmitted to an external devicesuch as the PACS 113 in accordance with the instruction for output.Determination is made regarding whether the image data instructed to betransmitted to the external device is a failed shot or not (Step S604),and further determination is made regarding whether or not to transmitto the external device (steps S607 through S610 and S613). An object foroutputting to the external device is generated based on the settingsexemplarily illustrated in FIGS. 13 through 14B (step S616), andtransmitted (step S617). Image data that is not a failed shot and issuitable for observation, for example, is transmitted to the externaldevice by the processing illustrated in FIG. 6. Assuming that thetransmission method of image data to the external device has been set to“transmit as supplementary information of base image” in the settings1308 for example, a DICOM image file 1201 having the ultrasound image(B-mode image) that is the base image is generated (FIG. 12). Thephotoacoustic image (oxygen saturation image) that is a layer image isincluded in the overlay image data 1206 included in this DICOM imagefile 1201. In another example, assuming that the transmission method ofimage data to the external device has been set to “transmit as referenceobject of base image” in the settings 1308, a DICOM image file 1201having the ultrasound image (B-mode image) that is the base image isgenerated (FIG. 12). Superimposing reference image information 1105(FIG. 11) corresponding to the oxygen saturation image is thengenerated. In a case where a color display of the oxygen saturationimage is desired, the superimposing reference image information 1105 isa CSPS object (omitted from illustration), for example. The CSPS object(omitted from illustration) has the same configuration as the GSPSobject 1207 illustrated in FIG. 12. The CSPS object (omitted fromillustration) has a SOP Instance UID corresponding to the base image forsuperimposing the oxygen saturation image as a layer image, i.e., theDICOM image file 1201 for the B-mode image. Accordingly, superimposeddisplay with the B-mode image as the base image can be realized at theviewer 114 based on the oxygen saturation image CSPS object (omittedfrom illustration). The user can easily perform superimposed display ofa suitable form.

Modifications

Although an example of generating an object of display at an externaldevice, in a form of suitable superimposed display of a plurality ofimage data acquired at the imaging system 100 in the above embodiments,the present invention is not restricted to this arrangement. Forexample, the above-described processing may be realized as a reportsystem. A report system is a system where the results of interpreting,where a physician observes image data of the subject and makes adiagnosis on the pathological condition and so forth of the subjectbased on the features instructed in the image data, are generated as aninterpretation report. Superimposed images, composite images, and datafor superimposed display, generated at the imaging system 100, are setas key images in an interpretation report by the above-describedprocessing, for example, thereby supporting the physician in generatingan interpretation report.

An example has been described in the above embodiment, where an objectfor displaying in a suitable superimposed display form at an externaldevice when outputting a plurality of image data obtained at the imagingsystem 100 to the external device, but the present invention is notrestricted to this arrangement. For example, an object for superimposeddisplay at the external device may be newly generated based on an objectfor output to the external device that has already been generated. Forexample, the control device 101 references a DICOM image file of aB-mode image already generated, and the user inputs operations relatingto superimposed display. The external transmission informationgenerating unit 127 identifies the image data for superimposed displaywith this B-mode image, based on supplemental information included inthe DICOM image file. For example, the examination in which the B-modeimage was taken is identified using at least one of the examinationinstance ID, series UID, patient ID, examination UID, and information ofdate and time relating to examination, included in the DICOM image file.The information of date and time relating to examination includesinformation of at least one of examination date, examination time, timeof acquisition of ultrasound image, and time of acquisition ofphotoacoustic image. The external transmission information generatingunit 127 selects image data for superimposed display with this B-modeimage from the series of image data acquired by the specifiedexamination. The examination control unit 125 registers informationrelating to this superimposed display in the superimposed displayhistory information 310 or post-processing history information 311 ofthe shot image information 302 relating to this B-mode image. In a casewhere an instruction to transmit the data relating to this superimposeddisplay to the external device is made by the user, the externaltransmission information generating unit 127 generates the object forthis superimposed display based on the above-described processing.

In the above embodiment, an example has been described where imageinformation to serve as a reference is selected in step S404, andthereafter a base image and layer image are selected in step S409 and asuperimposed image is displayed, as described with reference to FIG. 4,but the present invention is not restricted to this arrangement. Forexample, the selection of image information to serve as a referenceshown in step S404 does not have to be performed. In this case, acombination not included in superimposed image identificationinformation is selected from the correlated series of image informationobtained in step S403, and the flow advances to step S409.

In the above embodiment, an example has been described where informationto identify the superimposing order, base image, and layer image is usedas information for superimposed display, but this present invention isnot restricted to this. Processing for positioning a plurality of imagedata may further be performed, and information indicating positionaldisplacement among image data, and information indicating offset, may beused as information for superimposed display.

The present invention can also be realized by supplying a program thatrealizes one or more functions of the above-described embodiment to asystem or device via a network or storage medium, and one or moreprocessors of a computer in the system or device reading out andexecuting the program. The present invention can also be realized by acircuit (e.g., an application-specific integrated circuit (ASIC)) thatrealizes one or more functions.

The control device in the above-described embodiment may be realized asa standalone device, or may be realized in an arrangement where multipledevices are communicably combined with each other and execute theabove-described processing, and both arrangements are included in anembodiment of the present invention. The above-described processing maybe executed by a shared server device or server group. It is sufficientfor the control device and multiple devices making up the control systemto be communicable at a predetermined communication rate, and do notneed to be in the same facility or within the same nation.

An embodiment of the present invention includes a form of a softwareprogram that realizes the functions of the embodiment described above toa system or device, and a computer of the system or device reading outand executing code of the program that has been supplied thereto.

Accordingly, the program code that is installed to a computer to realizeprocessing according to the embodiment by the computer is in itself anembodiment of the present invention. Also, the functions of theembodiment described above can be realized by the processing where anoperating system (OS) or the like performing part of all of the actualprocessing, in the computer based on instructions included in theprogram that the computer has read out.

Forms obtained by appropriately combining the above-describedembodiments are also included in an embodiment of the present invention.

The present invention is not restricted to the above-describedembodiments, and various changes and modifications can be made withoutdeparting from the spirit and scope of the present invention.Accordingly, the following Claims are attached to publicly set forth thescope of the present invention.

The control device according to an embodiment of the present inventioncan display a desired image on a viewer.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. A control device comprising: acquiring means configured to acquire aplurality of images including an ultrasound image and a photoacousticimage taken in an examination; selecting means configured to select,from the acquired plurality of images, a first image, and a second imageto be displayed superimposed on the first image, based on informationrelating to the examination; and generating means configured to generatean object for outputting, to an external device, at least informationfor displaying the second image superimposed on the first image.
 2. Thecontrol device according to claim 1, wherein the acquiring means acquirea photoacoustic image generated based on photoacoustic waves occurringby irradiation of a subject body by light, and an ultrasound imageidentified based on information relating to the irradiation by light,and wherein the selecting means select the ultrasound image as the firstimage, and select the photoacoustic images as the second image.
 3. Thecontrol device according to claim 1, wherein the acquiring means acquirea plurality of photoacoustic images generated based on photoacousticwaves occurring in one irradiation of the subject body by light, andwherein the selecting means select the second image for superimposeddisplay on the first image from the plurality of photoacoustic images.4. The control device according to claim 1, wherein the selecting meansfurther select a third image for superimposed display on the first imageand second image, from the plurality of photoacoustic images, based oninformation related to the examination.
 5. The control device accordingto claim 4, wherein the generating means generate an object where anorder of superimposing the second image and the third image on the firstimage is identifiable.
 6. The control device according to claim 1,further comprising: accepting means configured to accept settingsrelating to generating of the object.
 7. The control device according toclaim 1, further comprising: correlation processing means, wherein theexamination includes irradiation of the subject body by light multipletimes, wherein the acquiring means acquire multiple photoacousticimages, each generated based on photoacoustic waves occurring in each ofthe plurality of irradiations by light, as the photoacoustic image,wherein the correlation processing means correlate, out of the acquiredultrasound images and photoacoustic images, an image group including aplurality of photoacoustic images generated based on photoacoustic wavesoccurring in one irradiation by light included in the plurality ofirradiations by light, and ultrasound images identified based oninformation relating to the one irradiation by light, and wherein thegenerating means generate the object for each image group that has beencorrelated.
 8. The control device according to claim 1, wherein theselecting means select based on information of time at which theultrasound wave signals and photoacoustic signals each were acquired inthe examination, as information relating to the examination.
 9. Thecontrol device according to claim 1, wherein the object includesinformation of at least one of examination instance ID, series UID,patient ID, examination UID, and information of date and time relatingto examination.
 10. The control device according to claim 9, wherein theinformation of date and time relating to the examination includes atleast one of examination date, examination time, time of acquisition ofthe ultrasound image, and time of acquisition of the photoacousticimage.
 11. The control device according to claim 1, wherein thegenerating means generate the second image as overlay image data as tothe first image, and generates a DICOM image file including the firstimage and the overlay image data as the object, as information forsuperimposed display of the second image on the first image.
 12. Thecontrol device according to claim 1, wherein the generating meansgenerate a DICOM image file including the first image as the object, andgenerate information for superimposed display of the second image on thefirst image as the object.
 13. The control device according to claim 12,wherein the object that is information for superimposed display of thesecond image on the first image that the generating means generate isone of a GSPS object and a CSPS object.
 14. The control device accordingto claim 1, wherein the selecting means select from images notdetermined to be suitable for diagnosis, out of the acquired pluralityof images.
 15. The control device according to claim 1, wherein thegenerating means generate the object including the first image.
 16. Thecontrol device according to claim 1, wherein the information fordisplaying the second image superimposed on the first image includes atleast information for identifying the first image.
 17. The controldevice according to claim 16, wherein the information for displaying thesecond image superimposed on the first image includes informationindicating an order of superimposing the second image, out of theplurality of images to be superimposed on the first image, on the firstimage.
 18. A control method comprising: acquiring a plurality of imagesincluding an ultrasound image and a photoacoustic image taken in anexamination; selecting, from the acquired plurality of images, a firstimage, and a second image to be displayed superimposed on the firstimage, based on information relating to the examination; and generatingan object for outputting, to an external device, the first image, andinformation for displaying the second image superimposed on the firstimage.
 19. A non-transitory recording medium storing a program thatcauses a computer to execute acquiring a plurality of images includingan ultrasound image and a photoacoustic image taken in an examination,selecting, from the acquired plurality of images, a first image, and asecond image to be displayed superimposed on the first image, based oninformation relating to the examination, and generating an object foroutputting, to an external device, the first image, and information fordisplaying the second image superimposed on the first image.